lopsa – azuen.net

The Whole World is Broken

In my dayjob, I’m a sysadmin. That’s a catch-all term that means I make computers work, but in reality, I make things work. By things, I mean everything. I’m the contact to the facilities manager, I paint walls sometimes, hang whiteboards from walls and TVs from ceilings, monitor and manage the cooling and power systems, and figure out office layouts. Oh, and I keep our network and computers running, which is actually in my job description, unlike all that other stuff.

I’m the only one in our small company who keeps those computers running, and the only one in our office with my particular skills and knowledge. You see, not all “computer people” are the same; I can write software, certainly, but not with the efficiency of those who do it full time; software developers can usually diagnose and fix problems with computers, but not as well as I can. There are subfields, and specialities; while a brain surgeon certainly knows a lot more about the heart than your average joe, if you’re having a heart attack, you’re better off with a cardiologist instead.

My field in particular, to great pain and displeasure, sometimes involves The Whole World Breaking. The causes of The Whole World Breaking are many, but the effect is the same: the entire company grinds to a halt, and everyone is unable to work or use the Internet or get email or print or do any of a thousand other things that makes a day productive.

And it’s on me to fix it.

When that does happen, I’m the only one in the company working. I’m the only one able to work. Everyone else is idle. As no one else has the specialized knowledge I do, and no one else is likely to be of help. No one else understands the whole system in its full complexity. But that doesn’t stop them from trying. People tend to discount the complexity of systems they don’t understand. They’ll therefore ask questions like “Have you tried unplugging the router and plug it back in!” as if a company network was just like a Linksys cable modem router in your living room.

Managers are the worst. They need to appear to be in control; that’s their title and rationale for existing, after all. They will crowd around, ask for continual “status updates” — which are always the same, to wit: “The Whole World is Broken.” Managers ask questions that have no answer: “What do you think caused it?” If I knew that I’d have fixed it. They don’t help, they just get in the damn way, and slow down the fix, which may be complex and intricate and require my full attention to address. A ten minute fix can therefore take an hour; and hour fix can take all afternoon. I spend a lot of time telling people that things are broken, trying to calm them down, nodding sagely that I understand they have deadlines and Important Business to take care of, and making completely bogus estimates as to how long it’ll take to fix the issue. And so I don’t end up with much time to actually, you know, fix the issue.

This pattern of behavior makes people feel more effective and more useful than what they should do, which is ordering me pizza and getting the hell out of the way. When I do Fix the Whole World, they’ll congratulate themselves for having Managed the Crisis. They feel involved, and feel of aid. It’s not a personality thing, either; I’ve never had a manager and co-workers who didn’t do this sort of thing, and my set of co-workers and managers have precious little else in common. It’s human nature to want to be useful in a crisis, so if they can’t be helpful, they try to appear to be helpful. Even when it’s harmful.

Now, I imagine, somewhere around the Gulf of Mexico, there is a small team of engineers who actually understand the Deepwater Horizon problem. There might even be that One Guy, the one that the other engineers and managers are looking to to Fix The Whole World. I don’t mean to be sexist, by the way: chances are very strong that it’s guy. That Guy intimately understands the risks, the complexities, and the difficulties of plugging the well. He lives in a world of uncertainties and probabilities, because complete understanding of complex systems is always a matter of uncertainty and probability. He understands that even when a certain thing should work, it sometimes doesn’t, and is left with choosing between a menu of potentially dangerous options to fix a damn well that’s been headline news for two months now.

And instead of dealing with just one manager and sixteen co-workers, he has to listen to journalists, the CEO of BP, a hundred million Americans, and Kevin frigging Costner standing over his shoulder and calling out suggestions. Because he’s The Guy, he’s the one called on for status updates, predictions and Congressional hearings, and is thus pulled away from actually fixing the well. Everyone else thinks the problem must have an easy fix, and that the engineer’s job is simple. The Joe the Plumbers of the world, with their down home common sense, think it should be simple — it’s just a big hole that needs to be plugged, isn’t it? How hard could it be — why not just blow the whole thing up? It’s a big issue, it’s in the news, and so after seeing one USA Today infographic, everyone’s suddenly an expert.

And if it’s easy to cap it, then clearly the trouble is that The Guy at the center of the storm is just lazy and stupid. So the correct response to flood BP and the engineering team with as many asinine suggestions and urgent demands as possible. That’ll light a fire under them.

Trust me when I tell you, there’s no one more motivated to plug that damn hole than those engineers. I’m sure they haven’t slept well since it started. Vacations have been canceled, childrens’ graduations have been missed. They’ve probably gotten a fair number of threatening letters sent to their homes. I’ll bet the CEO and VPs breathing down the engineer’s necks are the same ones who denied funding to safety measures the engineers recommended in order to prevent such a blowout, too. That’s how it works, in the world where professional management means that people with actual technical knowledge and a deep and personal appreciation of uncertainty and risk do not rise to decision making positions. Only people who understand so little that they do speak in absolutes and certainties rise to leadership.

Until they’re proven wrong, by a geyser of oil on the ocean floor. Or, for that matter, a flood of subprime housing losses. We’ve been here before.

And now the journalistic world thinks this is somehow Obama’s fault. I suppose they go with what they know, usually. Clearly these engineers will work faster if the President of the United States adds his voice to the jackasses already trying to micromanage a problem they don’t begin to understand. I’m not a huge Obama fan but I do give him credit for steering clear of the problem. It was probably actually helpful.

IT: Visionaries and futurists

The field of IT and system administration is infected by a lot of visionaries. It’s a relatively new field; by the most generous definition there have only been IT workers and sysadmins for 40 years. That makes computers and IT a sexy field that people who feel they’re so utterly brilliant that they can make livings predicting the future love to talk on and on about it.

There’s no shortage of visionary types. Lots of them end up in New York making and then losing piles of money. The financial markets are really nothing more than a huge gambling sport going on, in which every player thinks they understand the fiendishly complex game better than the other players. They’re right for a time and then tragically wrong. It’s no surprise they come to be wrong no matter how brilliant they are. After all, they’re making bets all over the place; one person is invested in Latin American mining interests, Chinese textiles, and European luxury goods. Another person might be in American software, Finnish timber, and Korean steel. How can anyone reasonably expect to understand all these industries well enough to even understand the present, never mind guess what’s going to happen?

They certainly don’t. So instead, financial analysts build models. They abstract out the way they expect a given industry works, and use that model to describe the characteristics of a company that should expect growth. They then use that model to try to find companies and investment opportunities that match those few magic qualities that make a sector successful, and bet their stakes — and other people’s money — on that. That’s all investing is; trying to distill what really matters in the face of the impossible task of fully understanding the wide world.

And it’s also why investing is perilous. Models are necessarily finite in their understanding; they only capture a subset of the data in an economy, and a subset of all possible conditions. They work for a time, if at all, but never forever. When the economy shifts underneath the model, the model’s assumptions might no longer be true, and investments made on it will turn out to be bad. If that happens to a particularly popular model, we have a Wall Street meltdown, where a whole lot of smart people look awfully stupid since they forgot at some point that their understanding of their own markets was extremely incomplete.

Technological futurists tend to be equally flawed, but they have the advantage that no one seems to care when they’re wrong, and keep buying their books anyway. It helps when you don’t have a billion dollar bet riding on your prophecy, I suppose. But lately Nick Carr has been the worst of the bunch; a deanship of the analysts that Rob Enderle ceded when he hitched his wagon to Microsoft and Microsoft turned out not to be the leader of anything anymore.

Carr is fond of telling us that IT is dead, and that computing power will become yet another utility; a power outlet, a hot water pipe, or the like; you plug in anything into any power outlet in America and the expected thing happens. There’s nothing that differentiates one type of power from another, one company from another, besides cost. And then power becomes a platform for other innovations, but it’s nothing that anyone should do on their own.

Carr makes the link that computing equals power, and then leaps further to say that a company can get no competitive advantage by innovating in IT, just like companies no longer have their own water turbines and gasoline powered generators. Any IT innovation is just going to be copied by your competitors and turn out to be null anyway. That means everyone has the same IT, and therefore it’s commodity; you can’t compete better with better electricity, and so you can’t compete better with better IT. So by all means, outsource whenever you can and concentrate on your “core” business, whatever that means, and wait for the day when email and computing power is standard and is in the “cloud.” This magical “cloud” will take over and rule all.

When people ask me what the computing cloud means, I never have an answer, because I have no friggin’ clue. I think it means he assumes Google is going to do all the programming for me. That would be nice, but I doubt it.

First, Carr’s hypothesis assumes an equal level of competence among company managements. Anyone who’s ever had a boss can tell you otherwise on that one. It also assumes equal talent among workers; also untrue.

Finally but most critically, Carr’s leading “insight” rests on a flawed assumption of what IT does. A lot of people see me and my compatriots as the tenders of the sacred shrine of the machine room; we know the rituals and incantations to Keep The Damn Things Running, and we should be measured in how well we do so. Lots of sysadmins see their role like that too. If that were the entirety of it, then yes, we could be outsourced I suppose to wherever you please, and computing would become a utility.

But keeping the damn things running is truly only the prerequisite to what we really do. My company, like most these days, is about knowing things. Knowledge gets turned into value and that gets turned into profit. These days, also, companies have much more data than fits inside people; so they’re in databases too vast to understand. They need to be processed and abstracted to derive the value. That’s what computing power does; processes data into knowable bits of things, taking the conditions of the world and trying to turn them into insight. You don’t truly know how to use a computer’s potential until you know how to program it.

The sysadmin is the gatekeeper. Our job is to funnel the business’s need to understand and use the data it already has, together with its need to keep it safe from disaster, secure from attack, and available on demand, into actual technology that can do these functions. Can that really be outsourced into an appliance? I doubt it; it’s a custom application. One type of database will not suit the needs of everyone, even within a narrow market. One type of communications medium may be perfect in one place and horrid in another. Everyone uses email, in a standard way, but no one does real business computing in a standard way, since there’s no abstraction that makes it general for all purposes.

Software programming is hard. It’s very abstract and fiendishly difficult to manage shifting requirements and changing ideas of what it should do. It’s hard because it’s an attempt to translate human desires — grey, ambiguous, difficult to communicate — into the concrete thought structures of software. Things that are that difficult and that intricate — and that tied and tailored to specific needs — are not easily made into commodities. Commodity implies one size fits all, but in this case, the detailed complexity of many problems demands one-off solutions.

As long as one-off solutions are necessary — as long as the nature of what I do changes dramatically whenever I take a new job, even if the description matches exactly the previous job’s — then there’s no opportunity for commodity economics. Indeed, people taking Carr’s advice and relying on “the cloud” will find that information — their most important resource these days — will be hard to find and harder to access.

And it’s no mystery to see how this happens. Carr’s a writer, not a sysadmin, and not an IT worker. He studies the problem but he doesn’t live it, just like a financial analyst on Wall St. He comes up with a model and applies it to the real problem, and generates headlines by making counter-intuitive pronouncements that you can get something that’s always been difficult — computing — for no effort and investment. He himself probably uses entirely commodity applications; can survive solely on a diet of Google mail and word processors and the like. But understanding doesn’t come from that; understanding comes from integrating details together; things that sound right can often be proven wrong when they come into contact with reality.

Expertise comes from the bottom up; knowing which factors lay in wait to trip up a big plan with a small snag, and being able to recognize a brilliant synthesis of a thousand details. In a way, if Carr were really analyzing, he never would have come up with his model and thesis; the data and analysis needed to provide a more complete vision of the IT industry would have been sufficiently difficult to assemble on computers that he might have recognized the role IT workers play in making that process easier.

So no, IT isn’t going away, nor should you want it. IT cuts can sometimes look good on paper, but the inefficiency drag that results in the rest of your business can easily set in and make your business sluggish, unresponsive and unaware even of what’s going on inside it, never mind in your market as a whole. The more expert your IT folks are in both technology but also your business itself, the more agile things will be. But if that bridge between technology and the business is never built, and you assume it’s just a commodity to understand your data, well, things will likely go very poorly.

Found day

So the snowstorm cancelled the Mardi Gras tournament Saturday, which resulted in a found day; an unexpected free day that I haven’t scheduled down to the hour. That was very nice, since it meant that I could spend it any way I wished, without a sense of waste or failure.

So I did. I watched a pair of movies, La Cite des Enfants Perdus, and the Wire DVD I had kicking around, from Netflix. The Wire’s incredibly good, and I blame Taco for making me watch it. The movie was suitably disquieting and strange in that odd French way. I also took a nap and did a metric ton of laundry. All in all, a day without computers and without commitment is rare enough that it should be appreciated as it comes. I only wish the outdoors were warmer and I had a garden to putter around or something of the sort.

Today I’m tweaking the tab software for NCFL registration and for some requests of a New Jersey persuasion. I’m running the NCFL paneling this year in addition to the tabbing, which means I have to strip out some dirty hacks and replace them with other dirty hacks. I also need to generate about four million different types of reports for various and sundry purposes. Nationals, as could be expected, thrives and runs on the triple-check. It’s the only way to manage a herd of that size.

But I finally managed to get the LOPSA website cleaned up. Now it’s readable and easier to look at by far. Next step is getting the menus and the navigation cleaned up, and perhaps at last get some writing going on there. System administration is a field ripe for a critical mass of thinking and writing; we’re written about a lot in my field, but we need to speak for ourselves now. It’s going to require a couple of hands pushing it forward, but if I have the time and the right forum, I think I can finally pull it off. Better late than never.

Valediction

One last day, and I am tired. I reflect on dinners with Chris, who alternates between being completely outraged by the things that are bothering me as well, and completely fed up. I admire his passion for it; I can’t really get that worked up about especially since I know I’m of limited means to do anything. I like a lot of things, and am having a great time even though I see troubles. Chris is a bit of a perfectionist, which both gives him the inability to enjoy things for what they are, but also much stronger ambition to change them.

Today is the game show, one of the highlights of LISA. I’m not of a mind to laugh at the moment, so I skip it, go back and pack my clothes for return on Saturday. The day is beautiful and the rest is welcome. I have a chance to re-charge, read a book a little while, play a game on the computer, watch the NBA preview for the weekend. The Patriots are playing Denver on Monday Night Football the next week.

I went to one last dinner with Chris, who had been somewhat abruptly ditched by people who probably didn’t notice what they were doing to him — and should have — and Skaar, who I hadn’t met until now. Skaar proved to be externally a gruff reserved Scandinavian and below that a man of sparking intelligence and humor with the most devastating one-liners of the conference, such as “Unfortunately, I don’t think he’s on drugs.”

The rack of lamb was marvelous, and we returned and hesitantly ended up in the party suite again. The crowd was subdued; everyone is over-LISAd. People drifted around and finished some alcohol; what is left could have still half filled the hottub. Whenever people started talking computers Patrick and I waved down the world. We piled on the couches, and I felt more secure and warm than I have all week. Patrick threatened to educate me using BeyoncÃ©. I forgave him his Motown inclinations. Michigan does that to people.

It was a marvelous finish. I realized I was on vacation even though I did everything I set out to for Harvard’s sake and my job. I had a fantastic time, but only because I met LISA on my own terms, and not its own. It has problems, and I am more sad at them than angry; the sadness of personal helplessness — I don’t have time to run for the exec, nor would I be likely to win election if I did. But where I am helpless others may not be so.

I got to know Chris in particular a good deal better, and met and got to know new people in JD, Mark, Patrick, and others. I saw Peg, and had more fun with Lois than I have in months. I got to have a great conversation at the booth with Toni and then failed to recognize her with her fake teeth and Halloween costume the next night. There were many people I knew going in that I didn’t talk to much, and that was all right, because there were many I did.

Perhaps I’m used to dealing with much more stressful travel and much more intense interpersonal dances — when people are aware of issues they tend to get talked about more. Perhaps suffering through years of badly run tournaments meant that I could approach LISA and not have to worry about the vortex of towering self-made significance that many others are overwhelmed by. I knew when to take a break and when I should give things up, like staying up late Thursday or the Game Show.

The last night, Friday, I didn’t sleep. We stayed up the night, and then Chris suggested as we were drifting off that we clean up the room a little, which I found admirable. I hope Geoff woke up pleasantly surprised that it wasn’t a disaster; he suffered much to provide us that space. Chris and Patrick and I wandered to an all night restaurant and ate a vague meal at 5 AM until we had to go back and prepare for our respective flights.

As we left the parking lot of the other motel with the family restaurant, a line of fire trucks six football fields long were lining up and leaving the parking lot in a sort of formation. The flag at the motels and hotels were flying at half mast, and all the lit signs were thanking the firefighters and police officers, more than 5,000 last I heard, who had fought the fire’s spread. They hadn’t quite won, but they had outlasted.

And as the dawn broke and we began to split our ways for our flights home, I saw a parade of names on the doors of the fire trucks. Las Vegas Fire Department. Flagstaff Fire Department. Grand Canyon National Park Service. Los Angeles Fire Department, despite having fires of their own. Phoenix Fire Department. Mesa, AZ. Bullhead Creek, NV. Many others I didn’t recognize.

A parade of names, little banners of pride in cities and states that had answered the call of a desperate city burning. They merged onto the interstate as a smoke-seeded rain at last pelted down, the first light of dawn bleeding through the clouds, and went home.

The horror of good ideas

By Thursday the wind finally had finished its shift, relieving San Diego of the smoke if not the fire. The technical sessions were interesting in their own right, though many were ill suited to my own job and my own domain. Many people grew hostile when I told them this simple truth; everyone things their solution is right for everyone and their assumptions are right for everyone. I’m amazed how many sysadmins can have the intellectual audacity to hold deeply seated opinions about infrastructure and especially security in the absence of any supporting data whatsoever.

We as a profession have a skewed sense of risk management; I suspect the world is not quite as hostile as we think. Most others design their systems so as to minimize failure and breakins; my systems, given our requirements, are designed instead to minimize the cost and effects of failure and breakins. It comes down to the same results, and I am both happy and well regarded in my choices by my small band of users engaged in their various unspeakable acts of computing. However, a horde of sysadmins is absolutely convinced that I will pay dearly for my approach, despite our computing being structured this way for eight years how, despite our empirical evidence accumulated over that time, and despite the fact that they can’t really point to anyone else or any other data that supports their conclusion.

The gap between potential and actual is very strong at these conferences. I listen to talks and think “Oh, good idea!” a thousand times, but I know of them only a handful will actually turn into good ideas. But that is enough. It’s good to think in the abstract, to plot ahead a little. Ironically, I do that most in the Hallway Track; outside of the talks, at lunch or in lounges talking with others about the world. I get more ideas, thing about more things.

That night, I go to sleep while it’s still Thursday. The parties apparently grew in intensity when I left. I’m happy to be unaware of it.