Simon St. Laurent
Untangling code with flow-based programming
That’s why we live in this world where we follow this one particular [von Neumann] architecture and all the alternatives were squashed… Turing gave us this very powerful one-dimensional model, von Neumann made it into this two-dimensional address matrix, and why are we still stuck in that world? We’re fully capable of moving on to the next generation… that becomes fully three-dimensional. Why stay in this von Neumann matrix?
Dyson suggested a more biologically based template-based approach, but I wasn’t sure at the time that we were as far from three dimensions as Dyson thought. Distributed computing with separate memory spaces already can offer an additional dimension, though most of us are not normally great at using it. (I suspect Dyson would disagree with my interpretation.)
Companies that specialize in scaling horizontally—Google, Facebook, and many others—already seem to have multiple dimensions running more or less smoothly. While we tend to think of that work as only applying to specialized cases involving many thousands of simultaneous users, that extra dimension can help make computing more efficient at practically any scale above a single processor core.
Unfortunately, we’ve trained ourselves very well to the von Neumann model—a flow of instructions through a processor working on a shared address space. There are many variations in pipelines, protections for memory, and so on, but we’ve centered our programming models on creating processes that communicate with each other. The program is the center of our computing universe because it must handle all of these manipulations directly.
Will WebRTC disrupt or be disrupted?
WebRTC promises to deliver computer to computer communications with minimal reliance on central servers to manage the conversation. Peer-to-peer systems promise smoother exchanges without the tremendous scale challenges of running video, for example, through central points.
The WebRTC Conference and Expo was unlike any other web conference I’ve attended. Though technologies in development are common at tech conferences, I can’t remember attending a show that was focused on a technology whose future had these levels of promise and uncertainty. Also, despite the name, WebRTC doesn’t resemble much of the Web despite being built into some browsers (more hopefully coming soon) and supporting HTTP(S) proxying.
The power of a technology now taken for granted
A friend wanted to show me a great new thing in 1993, this crazy HTML browser called Cello. He knew I was working on hypertext and this seemed like just the thing for it! Sadly, my time in HyperCard and an unfortunate encounter with the HyTime specifications meant that I bounced off of it, because markup couldn’t possibly work.
I was, of course, very very wrong.
Markup with some brilliantly minimal hypertext options was about to launch the World Wide Web. The toolset was approachable, easy to apply to many kinds of information, and laid the foundation for greater things to come.
Phil Dibowitz explains the challenges and the results they got with Chef
At OSCON, Phil Dibowitz reminded me how little I understand about large systems – as he puts it, really large systems, systems of systems, with some similarities but with different people controlling parts. His work at Facebook explores the challenges (and opportunities) of creating tools that work across a company’s many networks and computers.
If you deal with such challenges, he’s worth listening to as a model. If not, he’s worth listening to for a sense of just how different work at this scale can be, though much of what he accomplishes can be worthwhile at scales much smaller than the 17,000 servers he describes for Facebook at 26:42 in the session.
I talked with him in an interview:
and we’ve posted his OSCON session:
Flexible dreams for a responsive world
Last week’s Artifact Conference focused on the challenges of designing for multiple devices simultaneously. One frequent suggestions on stage and off was rough sketching, on screen or on paper, but it’s tricky to get there. The problem is easiest to see for designers on the Web, but it applies to a wide range of digital projects.
What does a sketch of a page look like when that page morphs itself to fit differently in different containers? What does a sketch of a site look like when that site may present different content (and different types of content) to different users based on their past encounters with the site? What does a sketch of a program look like when it handles many different kinds of input in many different kinds of circumstances?
One answer – the classic answer for those of us walking around with notebooks full of dot grid paper – is multiple sketches. Breakpoints become critical as “it looks like this when…” becomes a mantra.
At the same time, though, I rarely feel comfortable creating site (or code) details on paper when I know the result will be electronic. I spent way too much time explaining how paper and screens are different to saddened designers to want to inflict that pain on my own projects. My notebooks are largely filled with words, with occasional pictures.
Can version control manage content?
Web designers? Git? Github? Aren’t those for programmers? At Artifact, Christopher Schmitt showed designers how much their peers are already doing with Github, and what more they can do. Github (and the underlying Git toolset) changes the way that all kinds of people work together.
Sharing with Git
As amazing as Linux may be, I keep thinking that Git may prove to be Linus Torvalds’ most important contribution to computing. Most people think of it, if they think of it at all, as a tool for managing source code. It can do far more, though, providing a drastically different (and I think better) set of tools for managing distributed projects, especially those that use text.
Git tackles an unwieldy problem, managing the loosely structured documents that humans produce. Text files are incredibly flexible, letting us store everything from random notes to code of all kinds to tightly structured data. As awesome as text files are—readable, searchable, relatively easy to process—they tend to become a mess when there’s a big pile of them.
Can explanation contribute to technology creation?
“If you’re explaining, you’re losing.”
That gem of political wisdom has always been hard for me to take, as, after all, I make my living at explaining technology. I don’t feel like I’m losing. And yet…
It rings true. It’s not that programs and devices shouldn’t need documentation, but rather that documentation is an opportunity to find out just how complex a tool is. The problem is less that documentation writers are losing when they’re explaining, and more that creators of software and devices are losing when they have to settle for “fix in documentation.”
I was delighted last week to hear from Doug Schepers of webplatform.org that they want to “tighten the feedback loop between specification and documentation to make the specifications better.” Documentation means that someone has read and attempted to explain the specification to a broader audience, and the broader audience can then try things out and add their own comments. Writing documentation with that as an explicit goal is a much happier approach than the usual perils of documentation writers, trapped explaining unfixable tools whose creators apparently never gave much thought to explaining them.
It’s not just WebPlatform.org. I’ve praised the Elixir community for similar willingness to listen when people writing documentation (internal or external) report difficulties. When something is hard to explain, there’s usually some elegance missing. Developers writing their own documentation sometimes find it, but it can be easier to see the seams when you aren’t the one creating them.
Minimize assumptions, maximize testability
Model the flow of data instead of the logic of the program? That’s crazy! How can you encapsulate anything that way?
My piece on flow-based programming set off a lot of conversations, notably at Slashdot and Reddit. Many of them were hostile, wondering how such a different model could possibly work. Not all, but a lot, were the right kind of productive skepticism.
The reason this makes sense to me isn’t my time programming. My time in markup makes it seem much more sensible. While there are ways to incorporate content by reference (entities) in markup, parsers typically flatten those references into a single tree that is “the document”. Markup processing certainly can combine document transformations with data picked up from some other aspect of the program, but clean transformations are pretty ordinary.
In my broader programming experience, clean transformations are rare. Most of the code that I’ve written is all about creating logical components (often mixed with data) that communicate with other components. Data flows in fragments, and comes and goes from databases, other processing, and whatever structures seem convenient. Testing pieces in isolation is difficult, and worse, not always meaningful. If tests only apply when a program or process is in one state but not another, gaps will grow and test suites will expand perpetually.
The path I see forward – whether in functional programming, flow-based programming, or other attempts at sanity – is making state explicit at every step. This means structuring flows of information so that pure transformations are separated from side effects. Transformations take all the data they need directly as input, and generate only resulting data as output. Other functions (or processes, or whatever name they have in your system) handle all the interactions that have side effects. “Side effects” includes a lot:
Damian Conway customizes his favorite editor
Maybe you’re a hardened veteran of the editor war. Maybe you just need to make Vim do more for you. Or perhaps you just want to watch Damian Conway tell a great story with Vim as the central prop.
At OSCON 2013, Conway showed “Vim in the hands of a Real Maniac”. There’s very little I can do to add to his talk. Watch the whole thing if you’re up for a great performance, or explore the links below if you’re seeking out particular vim superpowers. He posted a tarball with the examples as well.
Flow-based, functional, and more
“Small pieces loosely joined,” David Weinberger’s appealing theory of the Web, has much to say to programmers as well. It always inspires me to reduce the size of individual code components. The hard part, though, is rarely the “small” – it’s the “loose”.
After years of watching and wondering, I’m starting to see a critical mass of developers working within approaches that value loose connections. The similarities may not be obvious (or even necessarily welcome) to practitioners, but they share an approach of applying encapsulation to transformations, rather than data. Of course, as all of these are older technologies, the opportunity has been there for a long time.
The Return of Flow-Based Programming
“There’s two roles: There’s the person building componentry, who has to have experience in a particular program area, and there’s the person who puts them together,” explains Morrison. “And it’s two different skills.”
That separation of skills – programmers creating separate black box transformations and less-programmery people defining how to fit the transformations together – created a social problem for the approach. (I still hear similar complaints about the designer/programmer roles for web development.)
The map-like pictures that are drawing people to NoFlo, like this one for NoFlo Jekyll, show how the transformations connect, how the inputs and outputs are linked. I like the pictures, I’m not terribly worried that they will descend into mad spaghetti, and this division of labor makes too much sense to me. At the same time, though, it reminds me of a few other things.