Transforming the Web (through transformation)

Decorating content may no longer be enough

Photo: http://commons.wikimedia.org/wiki/File:Metamorphosis_frog_Meyers.pngThousands of people invented it independently. Millions use it without thinking about a broader context. It’s time to name it so we can talk about it.

Transformation is changing the way we look at the balance between clients and servers, our approach to formatting and layout, and our expectations of what’s possible on the Web. As applications shift from transformation on the server toward transformation on arrival on the client, transformation’s central role becomes more visible.

“Templating” doesn’t quite capture what’s happening here. While templates are often a key tool, describing that tool doesn’t explain the shift from server to client. Templating also misses the many cases where developers are using plain JavaScript to insert, delete, and modify the document tree in response to incoming data.

These practices have been emerging for a long time, in many different guises:

  • In the Dynamic HTML days, scripts might tinker with the DOM tree as well as modify CSS presentation.
  • Transformation was supposed to be a regular and constant thing in the early XSLT plans. Stylesheets on the client would generate presentation from clean blocks of XML content.
  • Ajax opened the door to shell pages, apps that set up a UI, but get most of their content elsewhere, using JavaScript to put it in place.
  • New data format options evolved at about the same time that Ajax emerged. JSON offered a more concise set of programmer-friendly content tools. Many apps include a ‘bind JSON to HTML before showing it to the user’ step.
  • Template systems now run on the client as well as the server. In many systems, templates on the server feed data to the client, which applies other templates to that data before presenting it to users.
  • The HTTP powering Ajax still created a long slow cycle of interaction. WebSockets and WebRTC now offer additional approaches for collecting content with less overhead, making it easier to create many more small transformations.
  • Some developers and designers have long thought of the document tree as a malleable collection of layout boxes rather than a deliberately coherent base layer. Separation of concerns? A dead horse, apparently. Recent debates over CSS Regions highlighted these issues again.

Read more…

Comment |

A concrete approach to learning how to program

A solid foundation on which more meaningful learning can happen

578px-Perspectiva-1.svgAs someone who has previously taught computer programming for nearly a decade, I’m often asked questions that involve “what’s the best way to go about learning to program computers,” or “what’s the best way to get a software engineering job,” or “how can I learn to build mobile or web apps?”

Most of the readers of this blog have probably faced the same question at some point in their career. How did you answer it? I’ve seen many different responses: “come up with an idea for an app and build it,” or “get a computer science degree,” or “go read The Little Schemer,” or “join an open-source project that excites you,” or “learn Ruby on Rails.”

The interesting thing about these responses is that, for the most part, they can be classified into one of two categories: top-down approaches or bottom-up approaches. Top-down approaches are informed by the opinion that it’s better to be thrown in the middle of an application or a framework which encourages the learner to piece together knowledge in that context. Many books and online tutorials use an explicit top-down approach, often starting with the basics of a popular methodology, framework or technology. The most visible example of this are books on Ruby on Rails — they almost always inevitably begin with a description of the Model-View-Controller design pattern, but defer the incredible number of non-obvious ideas that make it up (Object-Oriented Programming, for instance).

On the other hand, a bottom-up approach starts with the basics/fundamentals of programming and then slowly builds your knowledge over time. In contrast to top-down approaches, bottom-up approaches try to minimize the number of these non-obvious ideas that the learner has to take for granted. Khan Academy and Code Academy are two examples of online sites that use a bottom-up approach to teaching programming. For the most part, they completely leave out any specific framework and focus on fundamentals of programming.

Read more…

Comments: 5 |

Implementing hypermedia clients: it’s not rocket science

Not ugly, not complicated


At Fluent 2013, O’Reilly’s Web Platform, JavaScript and HTML5 conference, Layer 7 Principal API Architect Mike Amundsen demonstrated how to build hypermedia clients, for situations with and without humans in the driver’s seat.

(If you’d like to know more about hypermedia in general, this interview provides more background.)

In his talk, Implementing Hypermedia Clients: It’s Not Rocket Science, Mike explored how hypermedia approaches drive conversation between clients and servers, and the application structures that result from those structures.

  • 1:44 – “The Semantic Gap: Hypermedia tells us what we can do, but it doesn’t say why.”
  • 6:04 – Hypermedia and application control information – links!
  • 8:09 – Control factors – “I accept RSS, can you give me RSS?”
  • 10:41 – Foundations of the class scheduling domain example
  • 16:30 – “What is a hypermedia client that a human would use?”
  • 19:24 – “Faithful Hypermedia Clients (FHCs) pass along whatever the server returns, and lets a human sort it out.”
  • 31:20 – “So what’s a Hypermedia for machine client?… Makes choices, not waiting for a human”
  • 33:25 – Working with Maze+XML
  • 37:10 – The power of generic types

If the Web Platform, JavaScript, and HTML5 interest you, consider checking out our growing collection of top-rated talks from Fluent 2013.

Comment |

Seduced by Markup

The power of a technology now taken for granted

A friend wanted to show me a great new thing in 1993, this crazy HTML browser called Cello. He knew I was working on hypertext and this seemed like just the thing for it! Sadly, my time in HyperCard and an unfortunate encounter with the HyTime specifications meant that I bounced off of it, because markup couldn’t possibly work.

I was, of course, very very wrong.

Markup with some brilliantly minimal hypertext options was about to launch the World Wide Web. The toolset was approachable, easy to apply to many kinds of information, and laid the foundation for greater things to come.

Read more…

Comment |

Walking Trees and Handling Events

The core of web programming, in JavaScript and beyond

This summer, I’ve seen all kinds of programming approaches as I’ve bounced between the Web, XSLT, Erlang, and XML, with visits to many other environments. As I look through the cool new possibilities for interfaces, for scaling up and down, and for dealing with data, I keep seeing two basic patterns repeating: walking trees (of data or document structure), and handling events.

Walking trees can be annoying, to put it mildy. The Document Object Model (DOM) is famously a headache for JavaScript (and other) developers. There are obvious opportunities for advanced developers to focus on graphs and other more flexible data structures as well. Trees are not necessarily the most efficient way to store information, especially when their content changes regularly.

Read more…

Comment |

Can We Do Better Than XML and JSON?

FtanML looks for the best of both

Today’s Balisage conference got off to a great start. After years of discussing the pros and cons of XML, HTML, JSON, SGML, and more, it was great to see Michael Kay (creator of the SAXON processor for XSLT and XQuery) take a fresh look at what a markup language should be.

Many recent efforts have been reductions. JSON was an extraction from JavaScript. XML was a simplification from SGML. MicroXML pushes simplification much further. Reductions are great for cleaning up past practice and (usually) making tools more accessible, but genuinely new features come later, if at all. The JSON and XML camps mostly stare at each other warily, and though people mix them, there’s little real “best of both worlds.”

Read more…

Comments: 10 |

Squeaky Clean Ajax and Comet with Lift

Focus on application development, not the plumbing

Lift is a web framework for Scala, and is probably best known for having great Comet and Ajax support.

I’ve been touring the features of Lift that I find appealing. Initially I looked at designer-friendly templates and REST services. Recently, I highlighted the great features for organising and controlling access to content.

Let’s now take a look at the Ajax and Comet features of Lift.


When I think of Ajax, I think of interacting with a web server, but avoiding page reloads.

An example: suppose we had a site that allowed you to write poems, and a feature on that site might be a button you could click to get an associated word from a thesaurus. The thesaurus we have is large, we don’t want it loaded in the browser, so we need to call the server to use it.

The HTML template would be a field and a button:

So far, that’s probably similar across many web frameworks. What you might now expect is for us to define a REST endpoint, do some jQuery wiring to connect the service to the button.

In Lift, we can do that, but often the Ajax goodness comes via this kind of Scala code:

This Associate snippet is binding the submit button’s click event to a Lift Ajax call. That is, the left side of the replace function (#>) is a CSS selector targeting the “onclick”, and the right side is arranging for the Ajax call to hit the server.

The first parameter to ajaxCall is the value passed from the browser to the server. We’ve asked for the value of the input field with an ID of “word”. The second parameter is the function to run when the button is pressed. Here we’re taking the word we’ve been sent, and updating the field with whatever our Thesaurus.association function gives us back.
Read more…

Comment |

Scaling People, Process, and Technology with Python

OSCON 2013 Speaker Series

NOTE: If you are interested in attending OSCON to check out Dave’s talk or the many other cool sessions, click over to the OSCON website where you can use the discount code OS13PROG to get 20% off your registration fee.

Since 2009, I’ve been leading the optimization team at AppNexus, a real-time advertising exchange. On this exchange, advertisers participate in real-time auctions to bid on individual ad impressions. The highest bid wins the auction, and that advertiser gets to show an ad. This allows advertisers to carefully target where they advertise—maximizing the effectiveness of their advertising budget—and lets websites maximize their ad revenue.

We do these auctions often (~50 billion a day) and fast (<100 milliseconds). Not surprisingly, this creates a lot of technical challenges. One of those challenges is how to automatically maximize the value advertisers get for their marketing budgets—systematically driving consumer engagement through ad placements on particular websites, times of day, etc.—and we call this process “optimization.” The volume of data is large, and the algorithms and strategies aren’t trivial.

In order to win clients and build our business to the scale we have today, it was crucial that we build a world-class optimization system. But when I started, we didn’t have a scalable tech stack to process the terabytes of data flowing through our systems every day, and we didn't have the team to do any of the required data modeling.


So, we needed to hire great people fast. However, there aren’t many veterans in the advertising optimization space, and because of that, we couldn’t afford to narrow our search to only experts in Java or R or Matlab. In order to give us the largest talent pool possible to recruit from, we had to choose a tech stack that is both powerful and accessible to people with diverse experience and backgrounds. So we chose Python.

Python is easy to learn. We found that people coding in R, Matlab, Java, PHP, and even those who have never programmed before could quickly learn and get up to speed with Python. This opened us up to hiring a tremendous pool of talent who we could train in Python once they joined AppNexus. To top it off, there’s a great community for hiring engineers and the PyData community is full of programmers who specialize in modeling and automation.

Additionally, Python has great libraries for data modeling. It offers great analytical tools for analysts and quants and when combined, Pandas, IPython, and Matplotlib give you a lot of the functionality of Matlab or R. This made it easy to hire and onboard our quants and analysts who were familiar with those technologies. Even better, analysts and quants can share their analysis through the browser with IPython.


Now that we had all of these wonderful employees, we needed a way to cut down the time to get them ramped up and pushing code to production.

First, we wanted to get our analysts and quants looking at and modeling data as soon as possible. We didn’t want them worrying about writing database connector code, or figuring out how to turn a cursor into a data frame. To tackle this, we built a project called Link.

Imagine you have a MySQL database. You don’t want to hardcode all of your connection information because you want to have a different config for different users, or for different environments. Link allows you to define your “environment” in a JSON config file, and then reference it in code as if it is a Python object.

Now, with only three lines of code you have a database connection and a data frame straight from your mysql database. This same methodology works for Vertica, Netezza, Postgres, Sqlite, etc. New “wrappers” can be added to accommodate new technologies, allowing team members to focus on modeling the data, not how to connect to all these weird data sources.

By having the flexibility to easily connect to new data sources and APIs, our quants were able to adapt to the evolving architectures around us, and stay focused on modeling data and creating algorithms.

Second, we wanted to minimize the amount of work it took to take an algorithm from research/prototype phase to full production scale. Luckily, with everyone working in Python, our quants, analysts, and engineers are using the same language and data processing libraries. There was no need to re-implement an R script in Java to get it out across the platform.
Read more…

Comment |

The Appeal of the Lift Web Framework

The extreme end of weird (as far as web frameworks go)

Lift is one of the better-known web frameworks for Scala. Version 2.5 has just been released, so it seems like a good time to show features of Lift that I particularly like.

Lift is different from other web frameworks (in fact, I labeled it at the extreme end of weird in the first presentation I gave about it), but people who get into Lift seem to love the approach it takes. It’s productive and enjoyable, which goes well with Scala.

I’ll keep this post short. Just two things:


You might be familiar with an MVC approach to the Web, where you have code that forwards to a view, and in that view you maybe use a little bit of mark-up to loop or display values. That’s not how it goes in Lift.

Instead, you start with the view first, and use HTML5 attributes to mark the parts of the view that need transforming. Here’s an example:

That’s valid HTML5. You can view it in your browser, or edit it in Adobe Fireworks, or whatever tool you want. The only part of it that looks a little strange is the data-lift attribute. What that’s doing is naming a Lift snippet, and a snippet is just a class. It might look like this:

What’s going on here? We’re using a nifty DSL in Lift to transform the <p> tag so that it contains the content we want. We’re saying…

Read more…

Comments: 3 |