Category: Expense Reduction

DevOps Alternatives

While many people involved in the tech industry have a wide range of experience in technologies and are interested in expanding the breadth of that knowledge, they do not have the depth of knowledge that a dedicated Unix support person, a dedicated Oracle DBA, a dedicated SAN engineer person has. How much time can a development team reasonably dedicate to expanding the depth of their developer’s knowledge? Is a developer’s time well spent troubleshooting user issues? That’s something that makes the DevOps methodology a bit confusing to me. Most developers I know … while they may complain (loudly) about unresponsive operational support teams, about poor user support troubleshooting skills … they don’t want to spend half of their day diagnosing server issues and walking users through basic how-to’s.

The DevOps methodology reminds me a lot of GTE Wireline’s desktop and server support structure. Individual verticals had their own desktop support workforce. Groups with their own desktop support engineer didn’t share a desktop support person with 1,500 other employees in the region. Their tickets didn’t sit in a queue whilst the desktop tech sorted issues for three other groups. Their desktop support tech fixed problems for their group of 100 people. This meant problems were generally resolved quickly, and some money was saved in reduced downtime. *But* it wasn’t like downtime avoidance funded the tech’s salary. The business, and the department, decided to spend money for rapid problem resolution. Some groups didn’t want to spend money on dedicated desktop support, and they relied on corporate IT. Hell, the techs employed by individual business units relied on corporate IT for escalation support. I’ve seen server support managed the same way — the call center employed techs to manage the IVR and telephony system. The IVR is malfunctioning, you don’t put a ticket in a queue with the Unix support group and wait. You get the call center technologies support person to look at it NOW. The added advantage of working closely with a specific group is that you got to know how they worked, and could recommend and customize technologies based on their specific needs. An IM platform that allowed supervisors and resource management teams to initiate messages and call center reps to respond to messages. System usage reporting to ensure individuals were online and working during their prescribed times.

Thing is, the “proof” I see offered is how quickly new code can be deployed using DevOps methodologies. Comparing time-to-market for automated builds, testing, and deployment to manual processes in no way substantiates that DevOps is a highly efficient way to go about development. It shows me that automated processes that don’t involve waiting for someone to get around to doing it are quick, efficient, and generally reduce errors. Could similar efficiency be gained by having operation teams adopt automated processes?

Thing is, there was a down-side to having the major accounts technical support team in PA employ a desktop support technician. The major accounts technical support did not have broken computers forty hours a week. But they wanted someone available from … well, I think it was like 6AM to 10PM and they employed a handful of part time techs, but point remains they paid someone to sit around and wait for a computer to break. Their techs tended to be younger people going to school for IT. One sales pitch of the position, beyond on-the-job experience was that you could use your free time to study. Company saw it as an investment – we get a loyal employee with a B.S. in IT who moved into other orgs, college kid gets some resume-building experience, a chance to network with other support teams, and a LOT of study time that the local fast food joint didn’t offer. The access design engineering department hired a desktop tech who knew the Unix-based proprietary graphic workstations they used within the group. She also maintained their access design engineering servers. She was busier than the major accounts support techs, but even with server and desktop support she had technical development time.

Within the IT org, we had desktop support people who were nearly maxed out. By design — otherwise we were paying someone to sit around and do nothing. Pretty much the same methodology that went into staffing the call center — we might only expect two calls overnight, but we’d still employ two people to staff the phones between 10P and 6A *just* so we could say we had a 24×7 tech support line. During the day? We certainly wouldn’t hire two hundred people to handle one hundred’s worth of calls. Wouldn’t operations teams be quicker to turn around requests if they were massively overstaffed?

As a pure reporting change, where you’ve got developers and operations people who just report through the same structure to ensure priorities and goals align … reasonable. Not cost effective, but it’s a valid business decision. In a way, though, DevOps as a vogue ideology is the impetus behind financial decisions (just hire more people) and methodology changes (automate it!) that would likely have similar efficacy if implemented in silo’d verticals.

 

What, me worry?

Steven Mnuchin, one of Trump’s best people, is not worried about mass worker displacement due to automation. Said so at an event hosted by Axios. I’d love some of whatever he’s been toking.

In the near term (and evidently that’s all business execs or government types care about these days), sure automation and AI will drastically increase profitability. But I foresee the trend following a similar path as off shoring … great for individual businesses, but at some point capitalism mandates people have some money to buy the stuff and neither offshoring or wide-scale automation is sustainable. Offshoring at least provided alternate jobs for enough people to float enough debt to sustain the market near-term. We’ve got “knowledge workers”. But what percentage of those can be turned into AI programs? A significant number. I automate 80% of IT work. Chat bots could provide at least half of legal and medical consultations — the routine stuff. Robots make products, load the truck/train/drone that drives itself. Right to your door, or even inside if you have the Amazon lock. There aren’t a lot of jobs where some portion couldn’t be automated today. And budget cuts and productivity demands essentially require it. Some lucky few own doomed companies and profit for some time, another really lucky few are AI programmers and electronics engineers (although self-building AI/robots are totally a thing too). Maybe automation will beget a whole new industry that will provide good jobs for billions of people. Maybe the capitalist system will collapse and everyone will have more than they need (the Star Trek series, I guess). But I don’t know that I wouldn’t worry about the impact automation has on employment and the economy.

Drone Army

Over the weekend, when it was negative five degrees, our neighbor’s power went out in the middle of the night. Some trees along the line grew into the power lines and had been abrading the line for some time, and a handful of arborists had to come out and try to trim the tree back. In the dark. At negative five degrees. Not the most fun job I could imagine, and the ironic this is it was the same team that had been out in the summer to clear trees along a stretch of the power lines a bit farther down.

The problem, it seems, is that it’s terribly time consuming to have arborists walking along the line to see where things actually need to be cut. Instead they just hit every section once per unit time. Sometimes that’s a quick couple branches snipped in a hardwood grove. Sometimes that’s serious maintenance in softwood groves. And sometimes delta-time is too long for, say, our line of pine trees. And sometimes the team doesn’t do a particularly good job of trimming the trees.

Made me wonder about having drones fly along the line – you’d still need someone to drive out, and I’d recharge the batteries in the van/truck so they’d be ready to go when I got to the next site. A single person flying a drone over a stretch of power lines could generate more realistic work orders for the arborists – skip the bits that didn’t grow much, realize these pine trees are endangering the lines before you had to call out a crew on Sunday night. They could also run through the same line post-maintenance and verify the work was done well.

Corporate Expense Reduction: Customer Service

We analyse calls to both our internal and customer-facing support desks all the time in an attempt to identify what causes the most calls. Theory being that fixing whatever-it-is would significantly reduce call volume and thus expenses. But do we ever analyze who is calling us? It may be a single problem that reoccurs. It may be ten little problems indicative of a bigger issue. It may be someone who is trying to avoid work and booking offline time on the line with the help desk (it’s been 20 years, but I used to generate a report for each call center manager of their director reports’ tickets & resolutions each week because it was so common a technique for a freebie 20 minutes off the phones and the manager didn’t have visibility into the ticketing system to determine if the calls were legitimate or not).

Multi-million dollar accounts (or senior company execs) get dedicated liaisons who walk issues through support channels. What if we had a small team of transient white glove support liaisons? People who have cultivated the same process knowledge and contacts that major accounts managers and exec support managers have. The top five callers for the week/month get rung up and their issue fixed. That may be calling around to get the proper support organisation looking at the issue. It may be replacing the thing that keeps breaking (replace the employee’s PC, replace the customer’s DLS modem). Even if you break even, there’s a much better experience with “yeah, I called in five times. But then so-and-so rang me up, took ownership of the issue, and fixed it” than “I have called eight times a week for the past month, had to explain the problem two dozen times, and these dolts finally managed to fix it”.

The other thing the metric could give you – and maybe a company would never use this – is how much a customer is *costing* the business. 800 charges, support staff man-hours … someone who has a 5$ a month account but uses ten hours of customer service time is costing the company money.

Corporate Expense Reduction: Energy

One of the things we’ve done with our home automation is tracking energy usage – partially because we want to size out a solar installation and the net metering in Ohio is awesome unless you produce more electricity in a rolling 12 month period than you use. So the installation has to be really close to your actual usage. But also because electricity costs money. Similar approaches may be beneficial to corporations. I’m using our 11 cent per kWh rate as an example. Actual rates depend on location and usage.

Does a company want to devote resources to “office automation” like we have home automation? Coupling motion detectors with smart outlets {or even just office schedules – if the last person is off shift at 7PM, dropping some device power at 8 should have no impact} to turn off power might save a lot in standby draw.

Even without home automation, companies can gather usage data to allow resources to be devoted to their biggest energy draws. The first step is identifying the big draws. We use Aeon Labs zwave clamp on home energy meters, but there are stand-alone energy meters. I’ve seen DIY Arduino based ones (https://olimex.wordpress.com/2015/09/29/energy-monitoring-with-arduino-and-current-clamp-sensor), or high end Fluke devices with clamps do the same thing (@5k+ for the Fluke … that’s a bit of an investment, but if someone’s got an energy metering capable fluke for other work ‘stuff’ … they may just need the 10$ clamps). Whatever equipment – clamp it on one circuit in a panel for a few days. Get a number, move it to the next circuit. Eventually you’ve got daily usage numbers for different circuits and just need to look at what is on those circuits to narrow down potential saving points.

We found obvious stuff – HVAC uses a lot of power. If a company leases a building with outdated equipment, use firm numbers in lease negotiation. The HVAC draws x kWh per year which costs us y $. A middle-road new system should draw z kWh which means we’re spending some concrete dollar figure per year because this system is so old. The same information can be used to cost-justify upgrades/replacements for company-owned buildings. Measure usage on lighting circuits. An office with old ballasts and florescent bulbs – what they are costing to run tells you if switching to LED {and there *are* LED T4/8 tubes that don’t require fixture replacements} makes any sense.

But we also found things I would never have even considered if I made a list of all of our non-trivial electrical draws. 20% of our annual electrical usage is the septic aerator (it literally uses more energy than the geothermal HVAC system in a year). We can get it down to 11% of our projected usage by cycling the thing on during even hours and off during odd (or on/off in twelve hour chunks, or 4 on / 4 off / 4 on / 4 off / 4 on / 4 off … new aerators have scheduling and do this themselves). Now that septic aerator savings is only like 250$ a year. Not a huge amount of money, but it’s 250$ I would never have realized we were spending otherwise.

From an IT perspective – if a server supports wake-on-lan … does a backup server and tape library need to be running 24×7? If someone kicks off a restore, can it be powered up (adds a minute, but saves power whenever restores aren’t running) and can it be programmatically powered on maybe half an hour before its backup jobs are scheduled to kick. Then power back down when no jobs report as running or scheduled for x hours. As a company, we mandate that all computers be left powered on so patches can be deployed overnight. What if the nightly patch check-in then powered the computer down (either because there are no patches or after installation in lieu of a reboot)?

Or a printer — there is no need for the printer to be in standby mode for the 15 hours a day no one is around to print. Or the weekends when no one is around. Or company holidays. One of the fairly large Xerox printers we have draws a continuous 11 Watts in sleep mode uses 71 kWh each year between 17:30 and 07:30 M-F and all day Saturday and Sunday. Maybe 72 kWh if you add company holidays. That’s not quite eight dollars a year in savings (and power consumption won’t be 0 if the device can be woken remotely) – but saving 6$ per printer in a company with 2000 printers is 12,000$ each year. Some of the older printers don’t even have a lower power sleep mode and draw 95 Watts in standby mode – 620 Watts per year when no one is around, and just under 70$ in electricity. Even better – HP offers an auto-off / auto-on on activity feature that allows energy to be saved during working hours.

Are there intangible benefits to energy saving initiatives? Get into the automation side of energy savings, would some tech magazine profiling the effort (free publicity, and tech magazines are a good place to advertise a company offering network services)? Can companies form partnerships with geothermal / solar / wind / whatever manufacturers to get cheap installations + publicity? Sadly, in some markets that may not play well (what, you don’t want to burn coal!?!) … but it might not be seen as a negative if it approached as a “save money, do right by stockholders AND customers” message instead of a “green, save the planet, global warming is bad” message.

Corporate Expense Reduction Strategy: Procurement

We have a 3D printer, and have an evolving list of things to 3D print. Custom design work comprises most of the list now. Originally, I anticipated printing a whole bunch of little plastic bits that are seriously overpriced in retail stores. The bobbins for embroidery thread come to mind — 20 cents a piece (although I’ve subsequently found them at Walmart for 10 cents a piece). They weigh practically nothing. 3D printer filament runs between one and a half cent per gram and four cents per gram (not getting into pricey exotic stuff which doesn’t make sense to use as a base for winding embroidery threads!). I can print six for Walmart’s super cheap price!

And then I thought to check AliExpress – two cents a piece. I can buy 100 for less than the price of 28 at Walmart. Material cost would be a little lower printing with the cheapest filament. But there’s electricity and time to consider as well. I ended up ordering the things shipped from China.

But in the process started wondering if companies use either of these techniques for reducing expenses. 3D printing would be an interesting endeavor – include the company logo, make items the *exact* size needed for an application. But direct ordering from overseas manufacturers has a larger opportunity for expense reduction. Pens for a cent or two each. Ten thousand staples for five bucks. Paper clips, wires and cords, clip boards … there are all manner of random little consumable things companies buy. Start running low on staples, order another ten thousand. So what it takes six weeks to arrive? You’re just refilling a supply closet.