- Starship — robotic delivery, from Skype co-founders. Pilot in the U.K. next year, in U.S. the year after. (via Brad Templeton)
- Materials that Couple Sensing, Actuation, Computation, and Communication (PDF) — very readable rundown of the ways in which materials can be designed to sense, compute, actuate, and communicate. You should read this because if the Internet of Things is going to be big, then the real breakthroughs and leaps forward will be in the Things and not the Internet. (via CCC Blog)
- Taiga — open source agile software project management tool (backlog, kanban, tasks, sprints, burndown charts, that sort of thing). (via Jef Vratny)
- Confidant — a secret management system, for AWS, from Lyft. If you build services that need to talk to each other, it quickly gets difficult to distribute and manage permissions to those services. So, naturally, the solution is to add another service. (In accordance with the Fundamental Theorem of Computer Science.)
"agile" entries
Four short links: 5 November 2015
Robotic Delivery, Materials Science, Open Source Project Management, and Open Source Secret Management
Four short links: 5 May 2015
Agile Hardware, Time Series Data, Data Loss, and Automating Security
- How We Do Agile Hardware Development at Meld — In every sprint we built both hardware and software. This doesn’t mean we had a fully fabricated new board rev once a week. […] We couldn’t build a complete new board every week, and early on we didn’t even know for sure what parts we wanted in our final BOM (Bill of Materials) so we used eval boards. These stories of how companies iterated fast will eventually build a set of best practices for hardware startups, similar to those in software.
- Recording Time Series — if data arrives with variable latency, timestamps are really probabilistic ranges. How do you store your data for searches and calculations that reflect reality, and are not erroneous because you’re ignoring a simplification you made to store the data more conveniently?
- Call Me Maybe, ElasticSearch 1.5.0 — To be precise, Elasticsearch’s transaction log does not put your data safety first. It puts it anywhere from zero to five seconds later. In this test we kill random Elasticsearch processes with kill -9 and restart them. In a datastore like Zookeeper, Postgres, BerkeleyDB, SQLite, or MySQL, this is safe: transactions are written to the transaction log and fsynced before acknowledgement. In Mongo, the fsync flags ensure this property as well. In Elasticsearch, write acknowledgement takes place before the transaction is flushed to disk, which means you can lose up to five seconds of writes by default. In this particular run, ES lost about 10% of acknowledged writes.
- FIDO — Netflix’s open source system for automatically analyzing security events and responding to security incidents.
What should replace the project paradigm?
Creating alignment at scale within enterprises.
The problems caused by using the project paradigm to delivering software systems are severe. The effect of projects on downstream teams such as release and operations were, for my money, most succinctly articulated in Evan Bottcher’s article “PROJECTS ARE EVIL AND MUST BE DESTROYED“. The end result — complex, heterogeneous production environments that are hard to change or even keep running — is due to the forces Charles Betz identifies in Architecture and Patterns for IT Service Management, Resource Planning, and Governance: Making Shoes for the Cobbler’s Children:
Because it is the best-understood area of IT activity, the project phase is often optimized at the expense of the other process areas, and therefore at the expense of the entire value chain. The challenge of IT project management is that broader value-chain objectives are often deemed “not in scope” for a particular project, and projects are not held accountable for their contributions to overall system entropy.
Furthermore, bundling work up into projects combines low-value features with high-value features in order to deliver the maximum viable product that is the inevitable result of the large-batch death spiral. This occurs when product owners try and stuff as many features as possible into the next release so they don’t have to wait for the one after in order to get them delivered. As a result, the median cycle time for delivering features is often poorly correlated with their priority — a highly undesirable outcome.
Why do we stick with it? Because our budgeting processes are designed to operate on projects, and project managers and the PMO know how to deliver them.
Since these are clearly poor reasons, what should we do instead?
What is DevOps (yet again)?
Empathy, communication, and collaboration across organizational boundaries.
I might try to define DevOps as the movement that doesn’t want to be defined. Or as the movement that wants to evade the inevitable cargo-culting that goes with most technical movements. Or the non-movement that’s resisting becoming a movement. I’ve written enough about “what is DevOps” that I should probably be given an honorary doctorate in DevOps Studies.
Baron Schwartz (among others) thinks it’s high time to have a definition, and that only a definition will save DevOps from an identity crisis. Without a definition, it’s subject to the whims of individual interest groups, and ultimately might become a movement that’s defined by nothing more than the desire to “not be like them.” Dave Zwieback (among others) says that the lack of a definition is more of a blessing than a curse, because it “continues to be an open conversation about making our organizations better.” Both have good points. Is it possible to frame DevOps in a way that preserves the openness of the conversation, while giving it some definition? I think so.
DevOps started as an attempt to think long and hard about the realities of running a modern web site, a problem that has only gotten more difficult over the years. How do we build and maintain critical sites that are increasingly complex, have stringent requirements for performance and uptime, and support thousands or millions of users? How do we avoid the “throw it over the wall” mentality, in which an operations team gets the fallout of the development teams’ bugs? How do we involve developers in maintenance without compromising their ability to release new software?
How to be agile with your big data
Agile methodology brings flexibility to the EDW and offers ways to integrate open-source technologies with existing systems.
Data analysis, like other pursuits, is a balancing act. The rise of big data ratchets up the pressure on the traditional enterprise data warehouse (EDW) and associated software tools to handle rapidly evolving sets of new demands posed by the business. Companies want their EDW systems to be more flexible and more user friendly — without sacrificing processing speeds, data integrity, or overall reliability.
“The more data you give the business, the more questions they will ask,” says José Carlos Eiras, who has served as CIO at Kraft Foods, Philip Morris, General Motors, and DHL. “When you have big data, you have a lot of different questions, and suddenly you need an enterprise data warehouse that is very flexible.”
EDWs are remarkably powerful, but it takes considerable expertise and creativity to modify them on the fly. Adding new capabilities to the EDW generally requires significant investments of time and money. You can develop your own tools internally or purchase them from a vendor, but either way, it’s a hard slog. Read more…
Four short links: 14 April 2014
dategrep, Agile Signoff, Feedback Speed, and Modern Dev
- dategrep — print lines matching ranges of dates. Genius!
- Business Case Guidance in Agile Projects (gov.uk) — how the UK govt signs off on Agile projects, which normally governments have no clue over how to handle properly.
- Hyper Growth Done Right — “While I was at Oracle, it took a month before a new engineer would get any code in,” Agarwal says. “It sent this implicit message that it’s okay to take a month to write some code.” First time I’d heard this wise point articulated: slow feedback loops send the message that progress can be slow.
- Docker + Github + Jenkins — clever integration of the three tools to get repeatable continuous integration. The modern dev environment has workflow built on git, VMs, and glue.
You Can’t Legislate Away the Time, Money and Features Law
More Lessons from the Collapse of healthcare.gov
Last week, I wrote in some detail about some of the specifics of how the Federal healthcare portal seems to have violated basic principles of good software delivery. Now I want to talk a bit about the more general factor that I think led to all those cut corners and shoddy deliverables.
One of the oldest and shortest jokes in the computer industry is “Time. Money. Features. Pick any two.” To some extent, you can swap quality out for features, because poorly delivered functionality is essentially non-existent functionality. In almost all commercial software projects, time and money both end up being the parts that give in the end. In other words, the original feature set almost never gets cut, but the project goes over budget and delivers late.
Intro to Raspberry Pi, Wharton Web Conference, Agile 2013, and More
Tech events you don't want to miss
Each Monday, we round up upcoming event highlights from the programming and technology spaces. Have an event to share? Send us a note.
The Revolution Will Not Be Televised webcast: Jonathan Stark discusses the coming wireless wave and how it will profoundly affect every aspect of society—the iPhone will look like a fax machine compared to what’s coming next. Register for this free webcast.
Date: 10 a.m. PT, June 20 Location: Online webcast
Intro to Raspberry Pi : Ed Snajder explains what a Raspberry Pi is, how it differs from an Arduino and shows attendees some cool things you can do with a Raspberry Pi. Register for this free webcast.
Date: 10 a.m. PT, June 25 Location: Online webcast
Four short links: 5 June 2013
Interesting Themes, Distributed Systems Failure Modes, Gesture Sensing Through Wifi, and Bad Taste Agile
- OATV Fund III Pitch Deck (Slideshare) — contains a list of what they were investing in, and what they want to invest in with the new round. Then: Quantified self; Internet subsystems; Smart networks of things; Manipulation and visualization of big data; sustainability; Maker movement. Now: Quantified Self Pro; Maker Pro; Hacking Education; Hidden Economies; Operations as Competitive Advantage; A Router in Every Pocket; The Internet Operating System. The move to “Pro” interests me, too. (via Bryce Roberts)
- The Network is Reliable — Many applications silently degrade when the network fails, and resulting problems may not be understood for some time—if they are understood at all. […] much of what we know about the failure modes of real-world distributed systems is founded on guesswork and rumor. […] In this post, we’d like to bring a few of these stories together. We believe this is a first step towards a more open and honest discussion of real-world partition behavior, and, ultimately, more robust distributed systems design.
- Wisee (PDF) — recognising gestures using disturbances in the (wifi) force. Our results show that WiSee can identify and classify a set of nine gestures with an average accuracy of 94%. (via BoingBoing)
- Why Your Users Hate Agile Development (IT World) — What developers see as iterative and flexible, users see as disorganized and never-ending. Here’s how some experienced developers have changed that perception. (via Slashdot)
Four short links: 6 April 2012
Agile FBI, Lucky Meat, Hiring Introverts, and Future of Reading
- FBI Uses Agile (Information Week) — The FBI awarded the original contract for the case management system to Lockheed Martin in 2006, but an impatient Fulgham, who was hired in 2008 to get the project on track, decided to bring it in house in September 2010. Since then, the agency has been using agile development to push the frequently delayed project across the finish line. The FBI’s agile team creates a software build every two weeks, and the pre-launch system is now running Build 33. The agency is working on Build 36, comprised mainly of features that weren’t part of the original RFP. Fulgham says the software is essentially done.
- Lucky Meat (Matt Webb) — the man is a mad genius. If you believe “mad” and “genius” are opposite ends of a single dimension, then I will let you choose where to place this post on that continuum. Then when you choose your tea (or coffee), the liquid is shot as if through the barrel of a gun BANG directly at your face. We use facial recognition computer chips or something for this. It blasts, and splashes, as hard and fierce as possible. And then the tea (or coffee) is runs down the inside slope of the “V” and is channeled in and falls eventually into a cup at the bottom apex where it finally drips in. Then you have your drink. (But you don’t need it, because you’re already awake.)
- Quietly Awesome — how are your hiring processes biased towards extroverts? See also I don’t hire unlucky people.
- How We Will Read (Clive Thompson) — Clive is my hero. I feel like we see all these articles that say, “This is what the e-book is,” and my response is always, “We have no idea what the e-book is like!” All these design things have yet to be solved and even thought about, and we have history of being really really good at figuring this out. If you think about the origins of the codex — first we started reading on scrolls. Scrolls just pile up, though. You can’t really organize them. Codexes made it easier to line them up on a shelf. But it also meant there were pages. It didn’t occur to them for some time to have page numbers, because the whole idea was that you only read a small number of books and you were going to read them over and over and over again. Once there were so many books that you were going to read a book once and maybe never again, it actually became important to consult the book and be able to find something inside it. So page numbers and indices became important. We look at books and we’re like, “They’re so well designed,” but it took centuries for them to become well-designed. So you look at e-books, and yeah, they’re alright, but they’re clearly horrible compared to what they’re going to be. I find it amazing that I can get this much pleasure out of them already. AMEN!