Automation Myths

The NSA Can't Replace 90% of Its System Administrators

In the aftermath of Edward Snowden’s revelations about NSA’s domestic surveillance activities, the NSA has recently announced that they plan to get rid of 90% of their system administrators via software automation in order to “improve security.” So far, I’ve mostly seen this piece of news reported and commented on straightforwardly. But it simply doesn’t add up. Either the NSA has a monumental (yet not necessarily surprising) level of bureaucratic bloat that they could feasibly cut that amount of staff regardless of automation, or they are simply going to be less effective once they’ve reduced their staff. I talked with a few people who are intimately familiar with the kind of software that would typically be used for automation of traditional sysadmin tasks (Puppet and Chef). Typically, their products are used to allow an existing group of operations people to do much more, not attempting to do the same amount of work with significantly fewer people. The magical thinking that the NSA can actually put in automation sufficient to do away with 90% of their system administration staff belies some fundamental misunderstandings about automation. I’ll tackle the two biggest ones here.

1. Automation replaces people. Automation is about gaining leverage–it’s about streamlining human tasks that can be handled by computers in order  to add mental brainpower. As James Turnbull, former VP of Business Development for PuppetLabs, said to me, “You still need smart people to think about and solve hard problems.” (Whether you agree with the types of problems the NSA is trying to solve is a completely different thing, of course.) In reality, the NSA should have been working on automation regardless of the Snowden affair. It has a massive, complex infrastructure. Deploying a new data center, for example, is a huge undertaking; it’s not something you can automate.

Or as Seth Vargo, who works for OpsCode–the creators of configuration management automation software Chef–puts it, “There’s still decisions to be made. And the machines are going to fail.” Sascha Bates (also with OpsCode) chimed in to point out that “This presumes that system administrators only manage servers.” It’s a naive view. Are the DBAs going away, too? Network administrators? As I mentioned earlier, the NSA has a massive, complicated infrastructure that will always require people to manage it. That plus all the stuff that isn’t (theoretically) being automated will now fall on the remaining 10% who don’t get laid off. And that remaining 10% will still have access to the same information.

2. Automation increases security. Automation increases consistency, which can have a relationship with security. Prior to automating something, you might have a wide variety of people doing the same thing in varying ways, hence with varying outcomes. From a security standpoint, automation provides infrastructure security, and makes it auditable. But it doesn’t really increase data/information security (e.g. this file can/cannot live on that server)–those too are human tasks requiring human judgement. And that’s just the kind of information Snowden got his hands on. This is another example of a government agency over-reacting to a low probability event after the fact. Getting rid of 90% of their sysadmins is the IT equivalent of still requiring airline passengers to take off their shoes and cram their tiny shampoo bottles into plastic baggies; it’s security theater.

There are a few upsides, depending on your perspective on this whole situation. First, if your company is in the market for system administrators, you might want to train your recruiters on D.C. in the near future. Additionally, odds are the NSA is going to be less effective than it is right now. Perhaps, like the CIA, they are also courting Amazon Web Services (AWS) to help run their own private cloud, but again, as Sascha said, managing servers is only a small piece of the system administrator picture.

If you care about or are interested in automation, operations, and security, please join us at Velocity New York on October 14-16. Dr. Nancy Leveson will be delivering a fantastic keynote on security and complex systems.


Sign up for the O'Reilly Programming Newsletter to get weekly insight from industry insiders.
topic: Web Perf/Ops
  • dsteakley

    i read the articles about this a few weeks ago, and it seemed to me that the statement about reducing sysadmins had been misunderstood. I don’t think the comments portended reducing staff. I think the comments suggested that there were too many people working in the intelligence community who had sysadmin rights, when they don’t really need them to do their jobs. Everyone’s seen this–it is much easier and faster to simply grant people unlimited access, instead of thinking through what access a specific person actually needs.

    • minstrelmike

      Actually, in a bureaucracy, it is almost impossible to define split access levels. The meetings are interminable and invariably, someone’s feelings are hurt. If you can’t give them a larger office, you can at least give them larger access ;-)

    • SystemHerder

      I agree with this assessment.

      My bet is that the NSA has been providing developers or support contractors administrative rights on the underlying systems where their applications or databases execute, and those rights have not been removed when they no longer need them. I doubt they are actually eliminating their 90% of their administrative heads, but actually 90% of the accounts that may have administrative rights to some system. I also expect that too many systems rely on the SAs to actually administer access, rather than integrating the system with a separate Integrated Access Management System and approval process.

    • Courtney

      The quote (via Reuters) from NSA Director Keith Alexander that most people (myself included) are using is this: “What we’re in the process of doing – not fast enough – is reducing our system administrators by about 90 percent.” While it’s not the clearest statement (e.g. “reducing” instead of “laying off”, which I ‘m sure was an intentional word choice on his part), it does seem to indicate a reduction in staff vs. changes in rights or permissions for existing staff.

  • minstrelmike

    Getting rid of excess sysadmins actually does increase security. Fewer people means fewer people-centered security holes.

    But if the sysadmins were excess to begin with, then it is security theater–which is what the American public wants (probably most publics).

    Bureau Administrators (as opposed to sysadmins) generally glom onto single-solution, one-size-fits-all kinds of schemes. Naysayers are demoted. People who say it won’t work are transferred.Then the ‘solution’ is implemented.

    As far as whether the solution improves ‘effectiveness’ depends on what is meant by effectiveness. That of course is defined by BAs (bureau admins) themselves which means if the solution is implemented, then of course it was effectively implemented.

    And that is how you end up with the ability to lose 90% of your sysadmins without losing any effectiveness whatsoever. ;-)

  • BK

    The politicians and managerial talking heads at the NSA are pathetically hilarious. I had a good laugh, anyway. Intending to cut 90% of your sysadmins is a clear ‘go ahead’ signal to all chinese, indian, brazilian, russian, and anonymous scriptkiddie wannabees to start pounding at the network. You had 1 guy leak info on purpose before. Now you will have entire world pounding at your infrastructure, not only reading but also writing information into it. Grab the popcorn, folks, this is gonna get even more hilarious.

    • jnojr

      I’m sure the entire world has been pounding on the NSA for years.

  • mpdehaan

    More consistent does mean more secure. However we must be careful, as automation software itself must also be secure rather than an extra attack point — a major focus of what we are doing with Ansible by just using SSH. There’ve been a lot of recent CVEs recently due to automation software and no one is immune.

    I expect much research in this in the coming future as we gain more systems, it is increasingly hard to observe what is going on *with* those systems. Unix by it’s very nature is a car with an open hood, SELinux goes to some extent to lock such data down (depending on policy), but the very nature of these systems means that increasing system count makes them harder to control.

    We should look at automation as a way to free up IT resources for more strategic thinking and doing a better job, rather than reducing headcount. Imagine if those resources were spent thinking of new ways to make things more secure, better backed up, more fault tolerant, etc — rather than cut. But you do want the number of people with access to the kingdom reduced.

    That all being said what files can be read by whom and transferred where *CAN* be controlled by computer software, and is kind of the point of an ironclad SELinux policy and similar concepts.

    Should the NSA cut access to who can limit what and require things to be done in parallel by multiple people? Absolutely.

    Part of the role of automation is to make IT processes doable in stage environments by a wider set of people doable in production by LESS people. It’s just like you don’t want your developers logging into prod when you are building AirBnbForCats.

    • DefineCommonSense

      That’s actually the silly thing about this whole mess with the NSA, mpdehaan. A low-level analyst made off with some of the kingdom’s treasure… and so who is getting punished? The serfs.

      The NSA isn’t punishing the security people that are in place to guard the information, but instead taking it out on the people responsible for maintaining the plantation they pull the information from. The fact that they think they can automate most of the work being done by 90% of their serfs is really just naive. They could be more realistic and shoot for 40% reduction, but still they are going to suffer productivity hits like what Nash is talking about.

      The only real way to even get 40% is to make sure that the departments that are going to be shouldering the layoffs have pristine documentation practices – and lets face it, there isn’t an IT organization on the planet that couldn’t do with “a little bit better documentation process”…

      And the NSA must really be hoping that whatever automation process they have in mind isn’t leaked, so that security researchers and script kiddies alike don’t start pounding away at it – because once that happens, they just won’t have enough serfs to alert the guards that the castle is under attack.

    • Bootang Tomakan

      More consistent means more consistent, nothing else. If the person designing and implementing the scripts does a good job, it will be more secure consistently, if they don’t it will be less secure consistently.

  • Lauren

    No one is going anywhere, but I do believe 90% of sysadmins will recontract as devops.

  • Nargunomics

    Surprisingly, I think there’s a decent analogy with boats. Once, boats could only be propelled by oars and paddles. To move a big boat, you needed many paddlers/rowers, but you could land on any beach. Then sails were developed, and you didn’t need the paddlers/rowers – you needed people to handle the sails instead. And a fully-rigged tall ship required as many rope-handling sailors as a Mediterranean galley required oarsmen. You also couldn’t land a fully-rigged tall ship on any old beach. You needed specialized locations and specialized staff. Then steam happened, and you needed less rope-handlers, but lots of stokers. And lots of miners, lots of docksiders, etc. Etc.

    If the NSA go through with this plan, they’re going to find themselves sticking their fingers in levees, hoping to hold back the raging flood, after letting the US HoS really piss the Chinese off with lectures on how it’s wrong to hack foreigners networks while the NSA etc were going full speed ahead hacking into China’s networks … May we live in interesting times … :)

  • Nick P

    My background is mostly on the security engineering and anti-subversion side of systems. I’m not sure NSA’s goal is doable. Here’s my ideas toward it, though, that I posted on another blog. I make the job high level, extensively use separation of duties, build in auditing, and automate wherever possible. It would still have risk and be a P.I.T.A., though.

  • Jean Habimana ,
    I found that website as interresting site and I suggest more people to take visits on it:

  • timkofu

    These are the guys who made Stuxnet. If they are determined to make an AI that manages the infrastructure, they probably can. Just a matter of time.

  • phuzz

    So who’s going to be maintaining and setting up the automated systems then? And who’s going to be monitoring them, and responding to the alerts?
    Unless the NSA’s sysadmins were spending 90% of their time doing nothing, this isn’t going to work. 10 people can’t do the work of 100.

  • deshantm

    You can’t replcae human creativity with computer automation.

  • justin

    I’m surprised the NSA was so badly run, this guy is mad at all the system administrators for their problem. Yet these are the very people than run and build their all important and complex systems 24/7. what an idiot.

  • otakucode

    It’s a retarded move. They are (it seems successfully) trying to shift the focus from the data they collect and analyze to the data that gets read by human beings. I don’t care what human beings read my email. I DO care (as everyone should) that they are using my emails to use machine learning algorithms to profile and detect ‘anomalous’ behavior. No human needs to be involved here. I don’t see why it’s ‘better’ to be profiled as a danger automatically by a computer than it is to have someone manually reading your communications. We need to stop them from collecting the data full stop. Not prevent them from reading it, prevent them from ever storing the data on any system which they control. And they think if they can get people to concentrate on the human element that they can get away with continuing to sweep up everything and analyze and profile it automatically.

  • Hazard

    There is no end of work for security, so there is no reason to imagine that fewer total employees will exist inside the secretive org. They just have different job titles. Assuming that they are like most federal employees, it can’t hurt to fire most of them.

    Automation does directly improve security planning. The former requires specific observation processes that increase the specificity of system tolerances. The increased resolution of understanding is a direct contributor to a focused security strategy, which in turn frees up time to do an even better job. I’m reminded of the old samurai maxim that “you don’t have to block every arrow shot at you, just block the ones that would hit.”

  • Mark Walker

    The authors of Operating Systems for the computers which followed mainframes did not bother to learn the hard won security lessons from mainframe-land.

    “In the mainframe era, IBM and its customers invested 15 years (1967-1982) building strong controls into computers, specifically to constrain the power of the systems programmers. System administrators are now as powerful as system programmers were in the 60s and 70s, and are unconstrained.” ~ Alan Paller, SANS

    It is no wonder security is now such a disheveled mess. This is what happens when security is not designed into systems. History has shown us adding security after the fact is a losing proposition.

    The NSA could maybe catch up with the USDA, eventually…

  • Ronald Pottol

    I used to work (quit in 2010) for a company that did management for large ecommerce sites, and it was quite disturbing how much was done manually. I’d bet that google has many more than 10x the number of systems per admin, and I can believe that the NSA was doing it all by hand.