"devops" entries

Four short links: 17 November 2015

Four short links: 17 November 2015

Remix Contest, Uber Asymmetry, Language Learning, and Continuous Delivery

  1. GIF It Up — very clever remix campaign to use heritage content—Friday is your last day to enter this year’s contest, so get creating! My favourite.
  2. Uber’s Drivers: Information Asymmetries and Control in Dynamic WorkOur conclusions are two-fold: first, that the information asymmetries produced by Uber’s system are fundamental to its ability to structure indirect control over its workers; and second, that Uber relies heavily on the evolving rhetoric of the algorithm to justify these information asymmetries to drivers, riders, as well as regulators and outlets of public opinion.
  3. ANNABELL — unsupervised language learning using artificial neural networks, install your own four year old. The paper explains how.
  4. Spinnakeran open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence.
Four short links: 12 November 2015

Four short links: 12 November 2015

Capsule Robots, Magnifying Deviations, Maker Books, and DevOps Theory

  1. Pillforge — open source software and hardware for Medical capsule robots aka cm-size mechatronic devices designed to perform medical tasks inside the body. Open sourced by Vanderbilt’s research team.
  2. Deviation Magnification — sweet image processing from MIT. Shares a researcher with this even more crazy paper on amplifying inconsistencies in rows of things. Mind: blown.
  3. Maker Humble Bundle — DIY bundle, pay what you want, optionally contribute to MakerEd.
  4. The O-Ring Theory of DevOps (Adrian Colyer) — Small differences in quality (i.e, in how quickly and accurately you perform each stage of your DevOps pipeline) quickly compound to make very large differences between the performance of the best-in-class and the rest.
Four short links: 11 November 2015

Four short links: 11 November 2015

Fundable Hardware Trends, Experience Heuristics, Robot Design Software, and Ops Feedback in Dev Tools

  1. 2015 Hardware Trends — HAXLR8R deck of the trends they see in fundable hardware.
  2. Heuristics — the heuristics and intuition risks that beset backcountry skiers are instantly recognizable to dev managers.
  3. Interactive Design of 3D-Printable Robotic Creatures (Disney Research) — paper describing software to let you design (add/remove motor-controlled legs, change shape, customize gait, etc.), modelling how they’ll move, and then 3D print when you’re happy. (via IEEE Spectrum)
  4. Runtime Metric Meets Developer: Building Better Cloud Applications Using Feedback (Adrian Colyer) — surfacing operations data like calls/sec, time to complete, etc. in the developer’s IDE. Wow, that’s genius. (And Adrian’s explanation/excerpts make this easy to digest)

Signals from the 2015 O’Reilly Velocity Conference in Amsterdam

Key insights from DevOps, Web operations, and performance.

People from across the Web operations and performance worlds came together for the 2015 O’Reilly Velocity Conference in Amsterdam. Below, we’ve assembled notable material from the event.

The Physical Web: A bridge between the Web and physical devices

The app-for-everything approach doesn’t scale, but the Web does. Scott Jenson, project lead for Physical Web at Google, outlines a vision for the Physical Web — an open approach to design and implementation that brings Web interaction to the physical world. “Let’s take the URL bar and bring it in the future,” Jenson says.

Read more…

Swarm v. Fleet v. Kubernetes v. Mesos

Comparing different orchestration tools.

Buy Using Docker Early Release.

Buy Using Docker Early Release.

Most software systems evolve over time. New features are added and old ones pruned. Fluctuating user demand means an efficient system must be able to quickly scale resources up and down. Demands for near zero-downtime require automatic fail-over to pre-provisioned back-up systems, normally in a separate data centre or region.

On top of this, organizations often have multiple such systems to run, or need to run occasional tasks such as data-mining that are separate from the main system, but require significant resources or talk to the existing system.

When using multiple resources, it is important to make sure they are efficiently used — not sitting idle — but can still cope with spikes in demand. Balancing cost-effectiveness against the ability to quickly scale is difficult task that can be approached in a variety of ways.

All of this means that the running of a non-trivial system is full of administrative tasks and challenges, the complexity of which should not be underestimated. It quickly becomes impossible to look after machines on an individual level; rather than patching and updating machines one-by-one they must be treated identically. When a machine develops a problem it should be destroyed and replaced, rather than nursed back to health.

Various software tools and solutions exist to help with these challenges. Let’s focus on orchestration tools, which help make all the pieces work together, working with the cluster to start containers on appropriate hosts and connect them together. Along the way, we’ll consider scaling and automatic failover, which are important features.

Read more…

Four short links: 7 October 2015

Four short links: 7 October 2015

Time for Change, Face Recognition, Correct Monitoring, and Surveillance Infrastructure

  1. The Uncertain Future of Emotion AnalyticsA year before the launch of the first mass-produced personal computer, British academic David Collingridge wrote in his book “The Social Control of Technology” that “when change is easy, the need for it cannot be foreseen; when the need for change is apparent, change has become expensive, difficult, and time consuming.”
  2. Automatic Face Recognition (Bruce Schneier) — Without meaningful regulation, we’re moving into a world where governments and corporations will be able to identify people both in real time and backwards in time, remotely and in secret, without consent or recourse.
  3. Really Monitoring Your SystemsIf you are not measuring and showing the maximum value, then you are hiding something. The number one indicator you should never get rid of is the maximum value. That’s not noise — it’s the signal; the rest is noise.
  4. Haunted by Data (Maciej Ceglowski) — You can’t just set up an elaborate surveillance infrastructure and then decide to ignore it. These data pipelines take on an institutional life of their own, and it doesn’t help that people speak of the “data-driven organization” with the same religious fervor as a “Christ-centered life.”
Four short links: 5 October 2015

Four short links: 5 October 2015

Semantic Sensors, Broadening "Sensor," Moving Fast, and Presidential Campaigns

  1. Semantic Sensors (Pete Warden) — tiny, cheap, all-in-one modules that capture raw noisy data from the real world, have built-in AI for analysis, and only output a few high-level signals.
  2. What if People Were Sensors, Not Things To Be Sensed? (Cory Doctorow) — Even in the Internet of Allegedly Free Things, humans and comput­ers are adversaries. Medical telemetry and implant companies envision selling shockingly intimate facts about your body’s internal workings to data-mining services and insurers. Car companies see their vehicles as platforms for gathering data on your driving, on traffic patterns, and on the sense-able facts of the streets you pass by, to sell it to, you guessed it, data-mining companies and insurers. John Deere has argued that its tractors are copyrighted works, and that it, not the farmers, own the soil-density data collected by the torque sensors on the wheels (it sells this data to Monsanto, which charges farmers for the right to know about it).
  3. Move Fast and Break Nothing (Zach Holman) — the first step is identifying what you cannot break.
  4. I’m Trying to Run for President But the Democrats Won’t Let Me (Larry Lessig) — A “democracy” in which 400 families give 50% of the money in campaigns is not American democracy. It is a banana republic democracy.
Four short links: 2 October 2015

Four short links: 2 October 2015

Automatic Environments, Majority Illusion, Bogus Licensing, and Orchestrating People and Machines

  1. Announcing Otto — new Hashicorp tool that automatically builds development environments without any configuration; it can detect your project type and has built-in knowledge of industry-standard tools to setup a development environment that is ready to go. When you’re ready to deploy, Otto builds and manages an infrastructure, sets up servers, builds, and deploys the application.
  2. The Majority Illusion in Social Networks (arxiv) — if connectors do something, it’s perceived as more popular than if the same number of “unpopular” people in the social graph do it. (via MIT TR)
  3. Scientist Says Researcher in Immigrant-Friendly Countries Can’t Use His Software — software to build phylogenetic trees, but the author’s a loon. It’s another sign that it’s unwise to do science with non-free software.
  4. Orchestraan open source system to orchestrate teams of experts and machines on complex projects.

Continuous Delivery versus Continuous Deployment

Download DevOps for Finance.

Download DevOps for Finance.

Download a free copy of DevOps for Finance, an O’Reilly report by Jim Bird for the financial services software insider who’s heard about DevOps, but is unsure whether it represents solution or suicide.

The DevOps Audit Defense Toolkit tries to make a case to an auditor for Continuous Deployment in a regulated environment: that developers, following a consistent, disciplined process, can safely push changes out automatically to production once the changes pass all of the reviews and automated tests and checks in the CD pipeline.

Continuous Deployment has been made famous at places like Flickr, IMVU (where Eric Ries developed the ideas for the Lean Startup method), and Facebook:

Facebook developers are encouraged to push code often and quickly. Pushes are never delayed and [are] applied directly to parts of the infrastructure. The idea is to quickly find issues and their impacts on the rest of the system and surely [fix] any bugs that would result from these frequent small changes.1

While organizations like Etsy and Wealthfront work hard to make Continuous Deployment safe, it is scary to auditors, to operations managers, and to CTOs like me who have been working in financial technology and understand the risks involved in making changes to a live, business-critical system.

Continuous Deployment requires you to shut down a running application on a server or a virtual machine, load new code, and restart. This isn’t that much of a concern for stateless web applications with pooled connections, where browser users aren’t likely to notice that they’ve been switched to a new environment in Blue-Green deployment.2 There are well-known, proven techniques and patterns for doing this that you can follow with confidence for this kind of situation.

Read more…

DevOps for Finance

How DevOps will help you surpass the common challenges of financial services software development.

Download DevOps for Finance.

Download DevOps for Finance.

Download a free copy of DevOps for Finance, an O’Reilly report by Jim Bird for the financial services software insider who’s heard about DevOps, but is unsure whether it represents solution or suicide.

DevOps, until recently, has been a story about unicorns. Innovative, engineering-driven online tech companies like Flickr, Etsy, Twitter, Facebook, and Google. Netflix and its Chaos Monkey. Amazon deploying thousands of changes per day.

DevOps was originally about WebOps at Internet companies working in the Cloud, and a handful of Lean Startups in Silicon Valley. It started at these companies because they had to move quickly, so they found new, simple, and collaborative ways of working that allowed them to innovate much faster and to scale much more effectively than organizations had done before.

But as the velocity of change in business continues to increase, enterprises — sometimes referred to as “horses,” in contrast to the unicorns referenced above — must also move to deliver content and features to customers just as quickly. These large organizations have started to adopt (and, along the way, adapt) DevOps ideas, practices, and tools.

This short book assumes that you have heard about DevOps and want to understand how DevOps practices like Continuous Delivery and Infrastructure as Code can be used to solve problems in financial systems at a trading firm, or a big bank or stock exchange. We’ll look at the following key ideas in DevOps, and how they fit into the world of financial systems:

  • Breaking down the “wall of confusion” between development and operations, and extending Agile practices and values from development to operations
  • Using automated configuration management tools like Chef, Puppet, CFEngine, or Ansible to programmatically provision and configure systems (Infrastructure as Code)
  • Building Continuous Integration and Continuous Delivery (CI/CD) pipelines to automatically test and push out changes, and wiring security and compliance into these pipelines
  • Using containerization and virtualization technologies like Docker and Vagrant, together with Infrastructure as Code, to create IaaS, PaaS, and SaaS clouds
  • Running experiments, creating fast feedback loops, and learning from failure

To follow this book you need to understand a little about these ideas and practices. There is a lot of good stuff about DevOps out there, amid the hype. Read more…