Transformative Programming

Flow-based, functional, and more

“Small pieces loosely joined,” David Weinberger’s appealing theory of the Web, has much to say to programmers as well. It always inspires me to reduce the size of individual code components. The hard part, though, is rarely the “small” – it’s the “loose”.

After years of watching and wondering, I’m starting to see a critical mass of developers working within approaches that value loose connections. The similarities may not be obvious (or even necessarily welcome) to practitioners, but they share an approach of applying encapsulation to transformations, rather than data. Of course, as all of these are older technologies, the opportunity has been there for a long time.

The Return of Flow-Based Programming

No Flo and its successful Kickstarter to create a unique visual environment for JavaScript programming reawakened broader interest in flow-based programming. FBP goes back to the 1970s, and breaks development into two categories:

“There’s two roles: There’s the person building componentry, who has to have experience in a particular program area, and there’s the person who puts them together,” explains Morrison. “And it’s two different skills.”

That separation of skills – programmers creating separate black box transformations and less-programmery people defining how to fit the transformations together – created a social problem for the approach. (I still hear similar complaints about the designer/programmer roles for web development.)

The map-like pictures that are drawing people to NoFlo, like this one for NoFlo Jekyll, show how the transformations connect, how the inputs and outputs are linked. I like the pictures, I’m not terribly worried that they will descend into mad spaghetti, and this division of labor makes too much sense to me. At the same time, though, it reminds me of a few other things.

graphic representation of flow through Jekyll

NoFlo diagram for noflo-jekyll

Punch Cards

Run a stack of punch cards through one program. Take the resulting cards and run them through another program. The programs don’t even have to be one same computer, so long as the second program understands as input the output of the first. The larger flow of the program might well be represented as a map of a physical delivery.

80-column punch card.

80-column punch card.

Unix Pipes

Sooner or later, every sufficiently fragmented system will look like Unix pipes, feeding the output of one process into the input of another (with a bit of buffering infrastructure). Unix pipes are another old technology. In an early pre-Unix memo, Doug McIlroy took inspiration from flow through a hose:

We should have some ways of coupling programs like garden hose–screw in another segment when it becomes when it becomes necessary to massage data in another way.

Unix pipes certainly make it easier for less advanced developers to take advantage of powerful tools created by experts. Despite that echo, though “less advanced” is still pretty advanced. Unix shell scripting still feels a lot like the programming used for the underlying components, and Unix culture and graphical programming may not be a common mix.

Functional Programming

Functional programming gives developers the opportunity to write their code as transformations all the way down to the bottom.

Clean functions – functions without side effects – are effectively pure transformations. Something comes in, something goes out, and the results should be predictable. Functions that create side effects or rely on additional inputs (say, from a database) are more complicated to model, but it’s easier to train programmers to notice that complexity when it’s considered unusual.

The difficulty, of course, is that decomposing programs into genuinely independent components in a fine-grained way is difficult. Many programmers have to re-orient themselves from orthodox object-oriented development, and shift to a world in which data structures are transparent but the behavior – the transformation – is not.

Much of what I loved about learning Erlang was that re-orientation. I could build complex data structures, but they didn’t have behavior. Every program was a series of transformations on data, and Erlang’s process structures and OTP made it possible to create intricate distributed pathways. Those structures are, of course, meant to be described by programmers and don’t likely fit neatly into the diagrams of flow-based programming.

Erlang pushes further than many functional languages in enforcing that separation between data and behavior. Developers coming to FP through, say, Scala or JavaScript, may not notice this distinction at first, but it’s a discipline I’d encourage them to embrace, a transition I’d encourage them to make.

Web Services and XML

When I first got into XML, I was thrilled by the open data aspect of it all. Developers could use whatever languages they wanted, and share information between them. The reality, unfortunately, was that such radical openness was terrifying. Companies leaped to get first-mover advantage through standardization, and in particular by standardizing tools for passing information between systems.

I can’t say that these tools quite looked like flow-based programming. Many of the early expectations came from CORBA and object-oriented programming as it was understood in the Java and .NET worlds. I somewhat regret writing about XML-RPC, which at first seemed like a relatively simple way to make a function call to an opaque service. SOAP and the WS-* specifications offered much more sophisticated possibilities. Business Process Execution Language in particular had some similar dreams.

On a simpler level, not attempting to distribute processing, XProc gives developers a set of tools for defining pipelines processing XML information. Basic control structures like for-each, choose, and try help developers pass information between transformations of many kinds. For some cases, XProc offers much the same opportunity as the latest generation of flow-based tools. XSLT is, by definition, a transformation language, and many other XML tools are effectively that as well. XProc makes it easy to connect them.

I’ve also pondered what role REST may play in these conversations. The PUT and DELETE methods are, of course, all about side-effects, and not so much about flow. I don’t yet have a good answer here.

(I also need to study the relationship between this, flow-based programming, and Jackson Structured Programming and Jackson Systems Development, which have come up in a few XML contexts.)

Humans as Transformers

So far, all of this sounds really automated. Programs pass around messages, sometimes over networks, and perform transformations. Humans might create inputs or deal with the outputs, but we’re pretty cut out of the conversation.

Humans, however, are extremely versatile if unpredictable transformers. We don’t just create information. We can edit, modify, and check it for sanity. While we may not want to see every step of every transformation – that’s why we write programs after all – there may be cases we do want to see. Error handling is a classic case even in highly automated systems, but flow-based approaches also mesh neatly with human workflows.

Most of the work I’ve seen in this space was built explicitly around human workflows. People create or edit something, put it into a system, and the system then modifies it and maybe passes it to another person or process.

However, even systems built with the assumption that they are primarily computing pipelines can incorporate human input. I was heartened to see Norm Walsh, for example, add a wait for (possibly human) update feature to the XML Calabash XProc tool in response to a request. (I suggested the same thing the next day, after it was already in!)

I’m also intrigued by the growing prominence of Git. While most people use Git as a source code management tool, its foundation is an extremely flexible toolset for recording transformations over time. Is that something that can be incorporated into flow-based programming models? I’m not yet sure, but I want to explore further.

The Dots aren’t Connected

My encounter with flow-based programming has me re-examining my assumptions about how to write programs and use computers. Many different aspects of computing seem to be converging toward that model, at many different scales and levels of granularity. It’s too soon to know whether this is a coincidence or a sign of things to come. I suspect, though, that we may finally be on the verge of a saner way to organize programs and programming.

Update: I should also note the Flow Based Programming Google Group for more conversation.

Related

Sign up for the O'Reilly Programming Newsletter to get weekly insight from industry insiders.
topic: Programming
  • http://www.deepFriedCode.com/ deepFriedCode.com

    From my experience, developers are looking for a new way to build applications. The difficultly we face with our old, tried-and-true methods is a sign that we’re doing it wrong. In working on my upcoming book “Dataflow and Reactive Programming Systems” I have come across many people that are also re-examining their assumptions. The difficult part is getting over the years of training that everything is an object. I am looking forward to a time when we use the best paradigm for the problem at hand, be it OO, functional or dataflow.

    • Simon St.Laurent

      There seems to be enough difficulty out there to make people think again! I’m not sure why I’m seeing it all now, but maybe I just wasn’t looking hard enough.

      I agree that “getting over the years of training” is difficult. Perhaps the piece that most intrigues me is the potential for letting developers use whatever style they like _inside_ of a transformation, while making the larger picture use a dataflow approach. That seems like a useful transition approach, at least.

      (I’ll admit to thinking that the functional programming folks can claim more overall consistency if we use FP or its cousins inside the transformations.)

      • Sean McGrath

        Todays’ FBP stuff reminds me a lot of what I was espousing to the XML community circa 2002. In particular the concept of “rigs” as networks of pipe-based transformations:

        http://xpipe.sourceforge.net/BinaryStuff/xpipeny.ppt

      • http://forresto.com/ Forrest O.

        New programmers (5 Billion potential programmers coming online in the next 10 years) don’t have to get over anything.

        I think that people want tools to solve problems. It is amazing to see the lengths that computer novices will go to get the wrong tools to do what they want. If we make it easier to stitch together minimal tools, then we could make our own environment for different kinds of tasks. Spreadsheet when we need tabular data, timeline when we have linear media, dataflow when we want to make A/V filters… building software should be the practice of recognizing how to stitch these components together.

        I think that the upcoming custom elements web standard + NoFlo is going to be a powerful tool to make tools to get stuff done. Making the larger (high-level) picture dataflow is exactly how I see this working.

      • http://www.deepFriedCode.com/ deepFriedCode.com

        In my opinion, dataflow languages of old were handicapped with the same thought process we had with OO… use dataflow from top to bottom. Flow Based Programming has the right mix. Use whatever paradigm you want for the nodes and layer the dataflow on top.

        As we are discovering about our universe (referring to space and planets and those things), there is a non-linear scaling from the smallest to the largest. That’s why Quantum Physics and General Relativity don’t mesh well together. In many ways, software mirrors the properties of the universe.

        If it is okay with you Simon… I invite everyone interested in Dataflow to take a look at my book at http://kck.st/198MGdt.

  • http://forresto.com/ Forrest O.

    I have been imagining a “Mechanical Turk” component for NoFlo: inputs data, text for directions, and price per item processed; outputs the human-transformed data. You could run these in parallel to compare answers, and use NoFlo’s normal flow controls to handle the asynchronous nature of the component.

  • Brendan Howell

    Flow based metaphors are really powerful if you’re dealing with discrete, independent transformation functions. Things like converting data from one format to another, live audio or video processing and simple routing systems can be modeled really nicely with flows.

    But as soon as you have any interdependencies and a need to keep track of state between functional blocks, it can become a real mess. It can get even more ugly once you start to have flows that feed back into the system. And lets not forget that graphical systems can be very limited in their ability to describe parallel or multi-instance structures.

    As an aside, you might want to check out PureData (PD), MAX-MSP and Labview for flow based coding systems. I know lots of people who love those tools.

  • GregorK

    Sticking to just one paradigm is also the problem. There are usually parts of the implemented system which are better described by state machine, others by flow, others by objects …etc. After years of experience I recommend to not be afraid to use a mixture of these paradigms and to use appropriate tools for them. It is no good to (re)implement state machine in java, it is better to use SMC (state machine compiler) for example.

  • Taylor

    Every tool has its use, I have worked in several different environments from industrial controls to consumer electronics.

    For a hardware control system for a robot or crash test sled I always use LabVIEW, because that’s what its for, controlling stuff that controls stuff. You can visually relate the flow sequence to the visual representation. I would love to see National Instruments embrace a more open approach to their software. I could very easily see it stagnating because the platform is so closed.

    For data abstraction and manipulating databases more “traditional” environments are still king.

  • davecb

    This reminds me somewhat of production systems (in the AI sense, not develpment vs production) and my least favourite program, sendmail.

    Discuss (:-))

    –dave

  • Joe Armstrong

    Hello Simon,

    Nice article. Glad to see you like FBP. There’s a really good book on flow based programming written by J.Paul.Morrison.

    The first edition of Morrisons book is at http://www.jpaulmorrison.com/fbp/. The book
    is fascinating and describes the work he did starting in the 1960′s. It’s well worth reading.

    Morrison talks of one application that went live in 1975 and is still running today.

    The NoFlo work seems to be based directly on Morrison’s ideas. I’ve been thinking for several years about implementing a back-end in Erlang :-)

    Cheers

    /Joe

  • Paul Morrison

    I often point people at something Steven Traugott wrote back in 2006 – http://www.jpaulmorrison.com/cgi-bin/wiki.pl?TheConvergence – in it, he listed a number of factors that he saw as converging to create major changes in our industry. To quote Steve, “Better hang onto your hats folks, this is going to be a wild ride…”

  • Alexander Golde

    NoFlo looks interessting, but is there anything like a debugger ? If it’s possible to integrate own pieces of JavaScript, how I can verify that they’re working properly ? Or do I have to create the program upfront without a chance to verify the results of the components?

    • http://forresto.com/ Forrest O.

      In the visual tool, we’re planning a few ways to do debugging. One way (live in the clock example) is to click on a wire to see the data flowing through. It is like probing an electronics project with an oscilloscope, and can be done in a running graph without changing modes. We’re thinking about a debugging mode that would artificially slow down the graph and show the data as it flows through. Traditional console printing and step-debugging work within nodes as well.

      • Alexander Golde

        Ok, got it. We at ANKHOR have implemented it similar. Will NoFlo only run in a browser then ? I think this could get quite challenging when you have to inpect larger data blocks. Our ANKHOR FlowSheet works on desktops only, one reason was this issue…

        • http://forresto.com/ Forrest O.

          The UI is separate from the running network, so it can communicate to another browser window (via postmessage) or server (via websockets). We hope to connect with other FBP implementations (go, python, php) through this protocol.

  • Joe

    “The Return of Flow-Based Programming”… I just…urg….

    There are at least 200k active licenses in LabVIEW. It’s working on 30 years old, and it’s used in dozens of industries.

    If you want to know where the problems in flow-based programming are, go troll around the NI site and lavag.org. Error handling, message structure, memory bloat due to copies, odd behaviors from in-placeness to avoid copies, shared state across boundaries for efficiency… there’s a pretty big list when you get into it. Big enough that there is a good community of professionals making a living solving those problems.

    Oh, well, maybe with the increased exposure of flow-based programming, I won’t get accused of “not being a real programmer” anymore.

  • samuel_lampa

    Really interesting article!

    I just had some thoughts about REST, to add to the conversation:

    The REST separation of the “actions” (GET/PUT/DELETE) from the “data model” (URLs), which used to be deeply entangled in SOAP like apis where you need to define a function for every data-item/action combination in non-standardized ways, is really powerful, IMO, but I suspect it makes most sense in the persistence layer, or public API of a data-system, than in the computation-, or processing layer.

    One could of course ponder the thought about using REST as a general data manipulation language for the so called Information Packets (IPs) also within a flow based process, but since the idea in FBP is to be able to treat the processes as black boxes, I guess that would any way be a question completely left to the implementer?

  • kombici

    klima servisi The Grandfather, Don Pacio Luciano, a reasonable, honourable man of integrity had sought justice after the deadly attacks on his family by suspected members of the klima servisleri notorious Sicilian Cosa Nostra. This is an outstanding story of love, honour, vengeance and astonishing baymak kombi servisi revelation of the secret operations……A note about those adjective-plus color names that women came up with but men didn’t: Women got those from department stores and drug stores. baykan kombi servisi Every one of those colors is credible as the name of a shade of eye shadow or lip gloss.I too kept my browser open for a long time when Time first came out. Alas, demirdökümkombi servisi time took it’s toll, and while I sometime demirdöküm servis s went back to check it, it gradually left my conscious demirdöküm soba servisieca kombi servisi