Month: September 2018

There is a difference

Today it is Jr’s “harassment” note, decades ago it was the assertion that “harassment is an ugly guy trying to get some”, but the fact remains that *harassment* there is a whole spectrum of harassment. Some dude whipping it out on the lunch queue, that’s blatantly obvious harassment without me specifically asking the individual to keep their genitalia covered. But harassment can be subtler too. The harassment I experienced frequently at work (twenty years ago) was the kind that became harassment when the guy refused to stop. A coworker asking me on a date is not harassment; a fellow student asking me to go to a dance is not harassment. Asking a dozen times *IS* harassment. Grabbing at my person and telling me how much I’ll enjoy the date *IS* harassment. And throwing them down on a bed, and attempting to disrobe them whilst covering their mouth … that’s not just harassment, that’s assault. And battery. And likely false imprisonment.

Agile Methodology Is Not Anarchy

For the past several years, my employer has been moving toward an Agile development methodology. There are some challenges when mapping this methodology into operations because it’s not the same thing; but, surprisingly, those are not where I have experienced challenges. The biggest challenge during this transition is some of my coworkers seem to think the methodology is that there are no rules.

A friend of mine, a fairly eccentric history professor, used to say that a little knowledge is a dangerous thing but you’ve got to emphasize LITTLE. And it seems like we’re encountering a situation where Phil’s emphasis holds true: the only thing garnered from from Agile training is that the documentation and process from waterfall projects are no more. But breaking away from the large-scale view of a P.R.O.J.E.C.T. for Agile development is a bit like breaking a monolithic application out into microservices — it still needs to do all of the same ‘stuff’, it just does it differently. And there are still policies and procedures — even a microservice team is going to have a coding standard, a process for handling merges, a way of scheduling time off, and some basic idea of what their application needs to accomplish. Sure, the app’s design will change incrementally over time. But it’s not an emergent property like chaos/complexity theory.

Maybe the “what Agile means to me” mentality comes from failing to clearly map a development methodology into an operations framework. Maybe it’s just a good excuse to avoid components of work that they do not enjoy. To avoid “agile operations” becoming “no boring planning stuff!!!”, I’ve outlined ways in which the Scrum methodologies the company wishes to adopt can be used to streamline operations. It helps that our group is reorganizing into an operational/support group and an architecture/design group — I see a lot of places within the operations team where Scrum approaches make sense.

Backlog — prioritizing the ticket queue like a backlog and having support staff constantly pulling from the top of the list — not only is this an awesome way to avoid the guy who scans the queue for the easy jobs, but it ensures the most important problems are being resolved first. A universal set of stakeholders does not exist for the ticket queue — someone whose ticket is ranked fifteenth on the list may disagree, and they are welcome to add details explaining why the issue is more impacting that it seems on its face. But 90% or more of our tickets are “Sev3” — which basically means both “we want it done ASAP” and “it isn’t a wide-spread high impact outage”. Realistically, dozens of tickets do not have the exact same time constraint and impact. There is extra work for management in converting a ticket bucket into an ordered backlog, but the payoff is that tickets are resolved in an order that correlates to the importance of the issue. In addition to the ticket queue, routine maintenance tasks will be included in the backlog. And prioritized accordingly.

Very short sprints — while developers moving from Waterfall to Agile might start from a month (or two) long sprint and trim weeks as they evolve into the process, operations starts from the other end of the spectrum. Our norm is to grab a ticket, sort it, then look at the queue and grab another one. We are planning for hours, maybe a day or two. This means we might establish application access on Tuesday that isn’t needed until next Monday. Establish a sprint that lasts a week, and use the backlog to get tickets that have lower priority (either because the impact is lower or because resolution is not needed for a week) included in the sprint. Service interruptions, SEV1 and SEV2 tickets, will occur and should be assumed in the sprint planning (i.e. either take enough work that you think it will just get done with no service interruption tickets and accept that some tickets from the sprint will be incomplete or leave some space for service interruption tickets and have staff pull “bonus” tickets from the top of the backlog if they have no work toward the end of the sprint).

Estimation — going through the tickets and classifying each incident as a quick little task, something that will take a few hours, or a significant undertaking facilitates in sprint planning. It’s difficult to know how many tickets I can reasonably expect to include in a sprint if I cannot differentiate between a three minute config change and a three day application rollout.

Multi-tasking — Implementation, support, and ticket resolution tasks are no longer a big bucket of work that individuals attempt to multi-task to complete. There are distinct tasks that are completed in series. Some tickets require information from the user; put the ticket on hold until a response is received and move on to the next unit of work.

Velocity — historic data based on time estimates cannot be generated, but simple number of tickets per week pre- and post- can certainly be compared. And going forward, ticket counts can be weighted by estimation values.

Stand-ups are a bit of a mental sticking point for me. I can conceive the value of spending a few minutes reviewing what you’ve done, what you plan on doing, and ensuring there is a ready forum to discuss any sticking points (maybe someone else has encountered a similar situation and can offer assistance). Stand-ups could include a quick discussion of any priority shifts (escalations, service interruptions) too. *But* my experience with stand-ups has been the attendance test variety — stand-ups that were used to hurt individuals who didn’t make it to the office by 08:00. Or those who weren’t around at 16:50. I don’t think it’s reasonable to ask someone who got into an issue and worked until 7P to show up at 8A the next day. I also don’t think it is reasonable to expect someone who came in at 6A to continue working until 5P. Were a stand-up scheduled in the middle of the day, I might feel differently about them.

Red Herrings

To everyone discussing whether a 17 year old kid who sexually assaults someone at a party (whilst high/drunk/whatever) should have said event preclude them from [promotions, government service, appointment to the Supreme Court] … that’s a red herring. The question is if someone who lies to Congress under oath (possibly repeatedly) should be confirmed to the Court. And that would be a resounding NO regardless of the individual’s politics.

There’s a big difference between elusive “I do not recall X” testimony where you’re not denying the action itself but rather recollection of said action and “I did not do X”. When someone pretty convincingly testifies that you did do X. Be X receiving stolen e-mails or sexually assaulting a woman … well, there are *lots* of things I’ve done but do not recall (although I’m not sure what kind of life you lead when stolen e-mails to advance judicial nominations are so every day that they simply slip your mind). But outright denying it happened?!?

History Without Context

There’s a challenge in teaching history to young people — whilst it is not good to proceed through life ignorant of what has come before you, there are facets of history that are simply incomprehensible to a five year old kid. Explaining why some people are afraid of the police, describing the point of the military … it is a snarl of sociological and political facts, individual experiences … there’s a good and a bad side, but it is difficult to understand points of view without the entire history that created that point of view (a bit like coupling Zinn’s People’s History with Johnson’s History of the American People and calling that a balanced history lesson). I used to advocate for the inclusion of fictional works in University history classes — while the story itself may not be true, fictional works provide a picture into the reality of the time. History provides a context for books, and books provide a context for history. Arthur Miller was not randomly enamored with the Salem witch hunts.

Sadly, Anya’s teacher has begun down the path of history without context. Today (why not yesterday!?!) she taught the kids that “bad people” crashed planes into buildings in DC and NYC, as well as PA. Which left me to try explaining that it’s not like half a dozen people woke up one morning and thought it might be a lark to try flying an aeroplane … only to find it wasn’t as easy as it looks on TV. It was an organized group executing a plan. It was also a group organized partially because of terrible things done across the globe. A cause can be just without justifying any action taken in support of the cause. The validity of a cause doesn’t make the action right any more than “he hit me first” makes slugging your brother right.

A lot of nation-states, countries, and people have done a lot of terrible things to one another in the name of just causes … the events of which the teacher spoke is an egregious example.

New Process Police

As an operational support group, we did not have a software development methodology. Doesn’t mean we didn’t develop software — one of the great things within operational support is the ability to automate day-to-day tasks to reduce workload. Why have someone check for application patches when a process can watch an RSS feed or file repository and notify us when there’s an update. Why have someone clickity-click provisioning users into groups when the user can make a web request, the group owner can approve the request, and an automated process can add the user into the group? The end result of our automation programming is, well, quite a bit of software.

And with a small number of people, informal application development worked. Wasn’t ideal, but it worked. If you want to write in Java while I use C# … not ideal, but the alternative is that one of us needs to learn a new programming language. Problem is the next guy we hired uses VBS, the next guy uses PowerShell … and I’ll use perl for simpler processes. Then someone starts tweaking my code and buggers it up … and we’ve got to figure out what happened and roll back based on some tape backup.

To get our internal software development processes organized, I developed a process. And ran a training session so everyone was familiar with both the process and the tools. Some of us have used the process well — don’t edit production code, clone the repo locally, make a branch for your edits, test it and have another group member sign off on the changes, merge your branch back into master, test more, then pull the code into production. The majority, it seems, have not followed the process at all. Changes are made to the code running in production, not incorporated into the Git repo. Six months after the new development process went in place, half of our code has improperly made changes!

To an extent, I consider this a management problem … if the department doesn’t want software development to be a free-for-all, then the department managers need to ensure their staff follows the process. If the department wants everyone to do their own thing — then get rid of the process and declare our methodology as “do whatever you want”. The challenge for managers, though, is that they don’t know that someone has edited code in production and failed to commit their changes into the repo. If only there were some way to watch for improperly edited code and alert us promptly.

Other scripts I’ve found to perform a similar function attempt to parse ‘git status’ to identify all sorts of issues — but that doesn’t address the specific problem that I’ve got. To facilitate identifying offenders, I wrote a quick Python script that searches a directory tree for git repositories and alerts us when changes have not been staged for commit. If you’ve staged the changes for commit, that won’t be identified. But the particular problem we encounter frequently … there are alerts for that.