Make magic with Ruby DSLs

Demystifying your favorite libraries' domain-specific languages

For better or worse, I believe you can develop basic, yet useful, applications in Ruby on Rails with just a minimum amount of Ruby knowledge. Rails tucks away details behind object-to-table mapping, routing, database preparation, and other necessities for web applications to function. So, is Rails magic? It may seem like something shady’s going on behind the scenes at first, but all of these examples are really just instances of well-designed domain-specific languages within the Rails framework.

A domain-specific language, or DSL, focuses on a particular problem set, or domain, instead of trying to be all things to all people. By contrast, typical programming languages like Ruby are general-purpose languages–they offer a large, varied set of tools to accomplish any number of tasks. Ruby itself is a great example of a general purpose language: You can use it to perform system maintenance tasks, retrieve data from external services, calculate statistics–not to mention, develop complex web applications. But what if you need to focus on a specific task, like running system backups, test-driving software development, or defining database migrations in a Rails application? This is where DSLs come into play.

There are two types of domain-specific language, as defined by Martin Fowler. An external DSL requires its own parser to process commands passed to it. The result is a language that will likely not look at all like the language it was implemented in. SQL, for example, is an external DSL. You interact with a database via a language developed specifically for creating queries–not in the language your database itself was written in.

An internal DSL, by contrast, leverages the features of an existing programming language and allows you to use them within the DSL. When you execute the DSL, you execute it within the constructs of the host language. As we’ll see in a moment, a number of common tools used daily by Ruby developers leverage internal DSLs.

Ruby is well-suited for this type of DSL. With a little knowledge of blocks and just a touch of metaprogramming, you can create Ruby-based internal DSLs with ease.

A basic example

As a simple introduction, let’s look at some code I wrote a few years ago to build classroom observation tools. These observations are called walkthroughs. A walkthrough gets used by a school administrator, who goes from classroom to classroom and records what she sees in a three-to-seven minute stretch. These small datasets are collected into a larger pool, which collectively paint a picture of how things are going on a school-wide level. Walkthroughs are typically recorded with a mobile device or tablet (though some schools still rely on paper forms).

Keeping the time constraint and form factor constraint in mind, we broke down the structure of a walkthrough as follows:

  • A walkthrough has many sections
  • A section has many item groups
  • An item group has many items, or individual things to watch for
  • An item can be recorded via a checkbox, a radio input collection, or short text input
  • An item optionally has one or more possible values, or possible responses to select

Following these rules, I designed the following DSL (excerpted from a larger walkthrough):

wt = Walkthrough.new
wt.walkthrough "Sample Walkthrough" do
  section "Classroom Information" do
    item_group "Subject" do
      item "Second Item", :radio do
        value "Math"
        value "Science"
      end
      item "Hybrid classroom", :checkbox
    end

    item_group "Content Overview" do
      item "Content being taught", :text
    end
  end

  section "Instructional Practice" do
    item_group "Vocabulary and Content Knowledge" do
      item "Builds student background knowledge", :checkbox
      item "Teaches content concepts", :checkbox
    end
  end
end

Looking at this example, we can see right away how a walkthrough is structured by the rules I just mentioned. We can also see that section, item_group, and item all accept blocks, and that item expects a symbol indicating the format of response it takes. It’s just a Ruby object! Above all, it’s easy to read and write–much easier than the original, YAML-based version it replaced!

Here’s the code I wrote to implement the DSL, slightly modified for presentation here. For the sake of demonstration, this simplified example returns structured HTML with basic, textual representations of form elements.

class Walkthrough
  def initialize
    @out = ''
  end
  
  def walkthrough(name, &block)
    div(name, :walkthrough, :h1, &block)
  end

  def section(name, &block)
    div(name, :section, :h2, &block)
  end

  def item_group(name, &block)
    div(name, :item_group, :h3, &block)
  end

  def item(name, format, &block)
    @out << "<div>"
    case format
      when :checkbox
        @out << "[ ] #{name}"
      when :text
        @out << "___ #{name}"
      when :radio
        @out << "#{name} "
      if block_given?
        instance_eval &block
        @out.chop!
      end
    end
    @out << "</div>"
  end

  private

  def value(name)
    @out << "( ) #{name} "
  end

  def div(name, type, tag, &block)
    @out << "<div class="#{type}"><#{tag}>#{name}"
    instance_eval &block if block_given?
    @out << "</div>"
  end
end

Nothing behind my magical little walkthrough DSL than a handful of Ruby methods! We define a few vanilla methods for walkthrough, section, and item_group. Things get a little more interesting when we skip down to the private div method, which gets called by these first three methods. Again, this is a simple Ruby method–but when it runs across a block, it runs instance_eval on it to run the code contained in the block. So when we include the following in our walkthrough definition:

 item_group "Vocabulary and Content Knowledge" do
    item "Builds student background knowledge", :checkbox
    item "Teaches content concepts", :checkbox
end

We create a new HTML div with an appropriately-sized header (<h3> for item groups), then evaluate the contents of the block. In this case, that’s a couple of checkbox items. Once the block has evaluated, we tack on a closing </div> tag to @out, and the method is finished.

Back in the class’ public methods, it looks like item gets some special treatment. In reality, it works the same way as div, with the exception that it includes a case statement to give unique treatment to checkbox, radio, and text item types. In reality, this could and likely should be refactored to improve overall code clarity. (I didn’t know about Sandi’s Rules when I originally wrote it.) I left it as-is to further show that, at its heart, this is still just Ruby.

These two features of Ruby–blocks and the ability to evaluate their contents on-the-fly–are what make it such a nice language for developing internal DSLs. Of course, you could take this even further, perhaps by converting Walkthrough to a module so it can be mixed into other Ruby objects, or applying some additional metaprogramming to reduce redundancy across the walkthrough, section, and item_group methods. As long as you’ve got a class with a well-defined API, though, Ruby should provide you with all the tools you need to convert it to a DSL.

With this DSL in hand, I now had a single, easy-to-read source from which to create the data structure to present a walkthrough form and store the data input through it. I could also use it to generate a printer-friendly PDF version, or even as the basis of a scanning solution to collect paper-collected data back into the central database.

Examples from open source

The previous example is basic by design, but in fact demonstrates much of what you’d need to reproduce functionality used everyday in Rails development:

  • ActionController’s before filter (before_filter prior to Rails 4) simplifies the process of defining methods to run prior to others. Practicing Ruby has a nice example of how to recreate this feature in a tutorial on domain-specific APIs.
  • ActiveRecord migrations simplify defining and manipulating SQL tables.
  • RSpec doesn’t ship with Rails out of the box, but is a common replacement for the built-in testing component. It’s also a classic, albeit complex, example of an internal DSL.
  • Looking outside of Rails, Sinatra is not itself a framework as much as it is a DSL for defining how to handle HTTP requests in a given application. Unlike the more complex examples listed so far, you can read through much of the DSL’s implementation in a single file.
  • Chef a tool for automating server management tasks like deployment and configuration, provides a friendly DSL for “cookbooks” defining repetitive tasks where attention to detail is important. Most importantly it provides a common language for developers and system administrators (“devops”), simplifying maintenance of complex infrastructures.

From the example I provided, and these examples from the wild, you might observe where DSLs are best-suited:

  • In tasks that require a lot of routine, abstractable functionality, such as dealing with HTTP (Sinatra) or talking to any variety of database backends (ActiveRecord)
  • In situations where source should be readable by people besides the developers (Chef, my code sample, RSpec to a degree)
  • Language that makes it easier to get stuff done as a programmer, and not more complex (all of the above)

I’ll leave exploring those examples as an exercise for the reader. Once you know the patterns to look for, it’s easy to spot a DSL in action–and from there, it’s just a matter of finding the library’s implementation of it to learn more and practice a little Ruby magic of your own.

tags: , , ,