J. Paul Reed
A “have-coffee” culture
Tending the DevOps victory garden.
Download Building an Optimized Business.
Download a free copy of Building an Optimized Business, a curated collection of chapters from the O’Reilly Web Operations and Performance library. This post is an excerpt by J. Paul Reed from DevOps in Practice, one of the selections included in the curated collection.
Any discussion surrounding DevOps and its methodologies quickly comes to the often delicate issue of organizational dynamics and culture, at least if it’s an accurate treatment of the topic. There is often a tendency to downplay or gloss over these issues precisely because culture is thought of as a “squishy” thing, difficult to shape and change, and in some cases, to even address directly. But it doesn’t need to be this way.
Sam Hogenson, Vice President of Technology at Nordstrom, works hard to make sure it’s exactly the opposite: “At Nordstrom, we value these different experiences and we value the core of how you work, how you build relationships much more than whether or not you have subject matter expertise. It’s a successful formula.” Another part of that formula, Hogenson notes, is the ethos of the organization: “It’s a very empowered workforce, a very decentralized organization; I always remember the Nordstroms telling us ‘Treat this as if it were your name over the door: how would you run your business and take care of your customers?'” [Nordstrom infrastructure engineer Doug] Ireton described it as a “have-coffee culture: if you need to talk to someone, you go have coffee with them.”
DevOps in the enterprise
Follow Nordstrom's journey to continuous delivery and a DevOps culture.
Would you open an email with a subject line of DevOps and pants? I’m not sure I would.
Six months ago I sent Rob Cummings an email with exactly that subject and he did. And we can be thankful he opened it, because by doing so, he invited us to look back at the fascinating history of Nordstrom’s implementation of continuous delivery and a “DevOps culture.”
The story begins in 2004, in a different era of web operations and performance. Back then, Rob and his team drove out to the colocation facility to deploy the e-commerce site. It was an era in which everything was a bit more heavyweight and things moved a bit slower. But that was OK, because most companies were still figuring the web out, almost as much as users were trying to figure it out.
Then the world started changing. Customer expectations changed. The business’ expectations changed. Heck, even developer expectations changed. As a leader in Nordstrom’s operations department, Rob had to adapt. And all of this was complicated by the fact that the increased pace was starting to strain his team and the systems he was responsible for maintaining. Read more…
Velocity NY recap
Operating on the edge and real-time performance emerged as key themes at Velocity NY.
Maybe it was the hustle and bustle of Times Square just within earshot. Maybe it was the smell of that legendary pizza on every corner. Or maybe it was having a front row seat in the financial capital of the world while the drama of a possible government default played itself out like a Broadway show. Whatever it was, Velocity New York felt markedly different than its west coast cousin.
Of course, Velocity’s inaugural east coast incarnation sported presentations on the state of the art of all the topics we’ve come to expect from the conference: nudging style sheets, hacking Javascript, battling memory usage, tuning database queries, and coaxing the network for every last ounce of performance on the Web. (And then doing it all over again for the mobile version…)
But discussions on scaling the operations of the squishy side of the organization–developing healthy team interactions, talking through the realities of the ever-elusive DevOps culture, and what it all means to scaling not just web sites but full-fledged businesses–were more present than ever.
Building for failure is a recipe for success
How you handle failure can mean the difference between "just another incident" and a revenue-stealing accident.
I was ready to get home. I’d been dozing throughout the flight from JFK to SFO, listening to the background chatter of Channel 9 as a lullaby. Somewhere over Sacramento, the rhythmic flow of controller-issued clearances and pilot confirmations was broken up by a call from our plane:
“NorCal Approach, United three-eighty-nine.”
“United three-eighty-nine, NorCal, go.”
“NorCal, United three-eighty-nine, we’d like to go ahead and…”
My headphones went silent, Channel 9 shut off.
I didn’t think too much of it as we continued our descent, flight attendants walking calmly through the cabin, getting us ready for landing. I had noticed our arrival path was one I was unfamiliar with, but nothing else seemed out of the ordinary… until we turned onto the final approach. In the turn, I noticed the unmistakable glint of firetrucks’ rotating red lights, lined up alongside the runway.
Starting Small with Great Expectations
Explicit expectations are key to operating at scale
Our lives are rife with expectations.
When we flip the light switch, we expect electrons to flow; when we issue CPU instructions, we expect to get the correct answer; when we look at commit logs in the source repository, we (hopefully) expect tests to accompany them and that our colleagues have run them, pre-checkin. But we’ve all probably been burned by these types of assumptions at some point.
In an operational environment, like the large scale websites and build farms we’re responsible for, these sort of expectations can be a costly cause of errors, and are one of the prime sources of miscommunication. Many a postmortem has uncovered that some expectation the ops team had of the development team was actually an assumption… and we all know that old saying about assumptions and donkeys.
Efficient, Effective Communication Still Often Elusive
In the operational environment, miscommunication can be costly; but there are some easy ways to improve it.
Editor’s note: This is part two in a four-part series on the “-ations” of aviation that can provide further insight into DevOps best practices and achieving them. Part one, on how standardization helps organizations scale and is actually a part of healthy DevOps culture, can be read here.
Communication is an enigmatic topic when it comes to engineering. Parts of our jobs—blueprints, chemical formulae, and source code—require extremely precise forms of communication (even if it doesn’t end up communicating to the steel, molecules, or silicon what we intended). But when it comes to email threads sifting through requirements, meetings about implementation styles and risk assessment, and software design documentation, we often fumble.
Let’s face it: there’s a reason the “engineer equals bad communicator” stereotype exists. But there are some simple things that can be done, both individually and technologically, to begin challenging that stereotype.
Dual Navigation Receivers Required
There are obviously many forms of communication. In an operational context, it’s useful to distinguish between static and active communication.
Process Is Not a Four-Letter Word
Standardization done right can save your sanity and improve your culture
Capital-P “Process” ™ is something many software developers, operations engineers, system administrators, and even managers love to hate.
It is often considered a productivity-killing, innovation-stifling beast whose only useful domain is within the walls of some huge, hulking enterprise or sitting in a wiki nobody ever reads.
I have always found distaste for process fascinating and now even moreso that configuration management and version control have become such core tenets of the DevOps movement. The main purpose of those tools is to provide structure for software development and operations to increase reproducibility, reliability, and standardization of those activities.