February 23rd, 2006

Brad @ Burning Man

Why I Don't Do Computer Security for a Living

Towards the end of my computer network engineering career, when it became unavoidably obvious even to me that my final manager at The Conspiracy was looking for any excuse to fire me, I spent about a year trying to manage some kind of internal transfer out from under him, to almost anywhere else in The Conspiracy. But one opening in particular came along that seemed, to me at the time, to be perfect for me. Some of you who read what I wrote over the last three days are probably wondering why I don't do just that for a living, now. The work is very important, it's obviously still interesting to me, there are companies who have real money to pay for expertise in this area, and I'm at the very least better than 99% of the people who do it for a living, if I do say so myself. The Conspiracy had just created a new department inside their Security division, and was looking for a manager of internal information security. I applied for the job, and I thought that the interview went swimmingly. But then, I always think that.

I didn't get the job. (Obviously, or I probably wouldn't be retired now.) I knew I had no guarantee of an answer, but I asked the guy who interviewed me why not me? His answer seemed bogus and cowardly to me at the time. In hindsight, though, I wonder if his boss was right. You see, both the director of computer security and the vice president over him agreed that of the two internal candidates that passed the initial interview, I was obviously the better qualified. Unlike the other applicant, I knew the subject backwards and forwards, inside and out. There was also no question that I was highly self-motivated, willing to work hard, and indisputably loyal ... to the Conspiracy. But the guy who would have been my boss's boss vetoed me, and here's what I'm told that he said, as best as I can remember it word for word: "Some day, I'll have to take whoever has this job with me into a meeting of the Board of Directors and other senior management. And if Brad gets the job, I'll have no idea in advance what he's going to say."

Consider, for example, one of the incidents from my job in LAN/WAN engineering that was brought up in my involuntary-exit interview. The sales rep for a company that I won't shame by name (but you've heard of them) had wrangled an interview with my boss, the manager of all internal support operations, to sell him a network monitoring tool. The particular sales pitch for this tool was that the same tool, using the same reporting protocols, could run on everything we had: both flavors of mainframe, both flavors of minicomputer, all three LAN file server operating systems, and all three of our desktop operating systems. It would then compile uniform up-time statistics, and do unified alert reporting via either LAN or mainframe console. In particular, he was pitching this as a solution to our existing problem that the company had 24x7 mainframe operator staff, but wouldn't staff LAN administration 24x7; with this tool, LAN outages would show up on the mainframe operators' consoles. He brought with him a lengthy favorable review of his product that had appeared in a glossy industry news weekly which, again, I won't shame by name (but I guarantee you've heard of them).

By the time the sales rep left, my boss was 100% sold on this idea. So he brought me in, and rather than tell me this, he asked me if I'd ever heard of the product and if so, what I thought of it. I told him that I knew it well, that we'd used an earlier version of it at one of my previous employers ... and that it was an unreliable heap of garbage. I told him that it was flatly incompatible with our LAN protocols, that contrary to what they said it was incompatible with one of our two more important LAN operating systems and that the support for the even more important one was brand new and still in beta test. I also told him that at our previous company, which was even bigger than The Conspiracy, we had exactly zero luck getting their tech support to fix any of the mission-critical bugs we reported to them. I also pointed out to him that in order to work, he'd need to get the mainframe components of it installed, and I knew for a fact that since we had no meaningful test bed for that install, the managers who would have had to sign off on it were never going to even put it into test, let alone go live with it. He told me that the salesman said otherwise; I responded that of course he'd say that, he's paid via commissions. He showed me the news article. I told him that that particular rag will print any vendor's press release as a review, complete and unedited, with no testing of their own, so I wasn't terribly surprised that the review agreed with the salesman. And only after I'd said all of that did he tell me that he'd already put in the order for the software, and that getting it working on our test-bed system was now my job.

I told him that they paid him to make these decisions, not me, and I understood that. I told him that if anybody could make it work, it would be me, and I'd give it my flat level best attempt ... but that I made no guarantees, because I had no idea how I was going to get it to work around the known incompatibilities, and that I was going to be completely dependent on him to get the approvals from upper management to install the mainframe components. Which he never did. Nor, as I predicted, did the vendor fix any of the glaring bugs in the beta test part of their product; to my vast disgust, their network monitoring component, the thing that was supposed to track network reliability, was the single least reliable thing on our network. And that's where the project stood when I was fired on trumped-up charges, and one of the things that was brought up in my exit interview was that my boss suspected me of sabotaging an important company project, of intentionally sabotaging the network monitoring software he'd bought in order to make myself look smart and him look dumb.

If I were in charge of Internet security for a Fortune 500 company, I can already see how it would go. Some software vendor like Microsoft or SAP or Symantec or even Apple would be in there with some multi-gazillion dollar proprietary investment in their products that they wanted us to make, in order to protect our network from hackers. I would point out that even if their software and hardware solution worked as advertised, it would do exactly nothing to solve any of our more serious problems, that the money would be better spent on hiring smarter people and training the people we have better so that they don't fall for every phishing scam, social engineering attempt, and emailed trojan horse that came down the pipe, and bring in infected stuff from home to bypass anything we had put in place to block infections. I would call the current industry fad Flavor of the Month in network security solutions the Emperor's New Clothes, and compare investing in it to re-arranging the deck chairs on the Titanic.

And soon as I walked out of that meeting, the other senior managers would start chattering about who in the heck did I think I was, disagreeing with every glossy business and popular technology rag like BusinessWeek and Computerworld and Wired? The salesman/consultant seemed much more likable and seemed to know his stuff much more than I did; look how much the salesman/consultant, after all, understands business better (dresses and talks exactly like the other senior managers do) and is obviously much more up to date on the technology than I am (parrots back the same stuff they read in magazines). It would go unsaid, but also be true, that the consultant and/or salesman would be some golfing buddy of the CEO, or member of his club, or relative who needs the money, or some such thing. It would also be pointed out that it's more important to reassure the shareholders that we're doing what we should be, by doing what everybody else is doing, than it is to actually solve the problems, that there's no particular advantage in being right when everybody else is wrong. And in not much more than two years, tops, I would have been encouraged to pursue other interests.