I want to build a web server, a mail server, a BitTorrent client, a DNS server, or an IRC bot—clients and servers for a custom protocol in Python. And I want them to be cross-platform, RFC-compliant, testable, and deployable in a standardized fashion. What library should I use?
Use Twisted
Twisted is a “batteries included” networking engine for writing, testing, and deploying event-driven clients and servers in Python. It comes with off-the-shelf support for popular networking protocols like HTTP, IMAP, IRC, SMTP, POP3, IMAP, DNS, FTP, and more.
To see just how easy it is to write networking services using Twisted, let’s run and discuss a simple Twisted TCP echo server:
from twisted.internet import protocol, reactor
class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
class EchoFactory(protocol.Factory):
def buildProtocol(self, addr):
return Echo()
reactor.listenTCP(8000, EchoFactory())
reactor.run()
With Twisted installed, if we save this code to echoserver.py and run it with python echoserver.py, clients can now connect to the service on port 8000, send it data, and get back their echoed results. For example, using the telnet command:
$ telnet localhost 8000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. hello hello goodbye goodbye
This example uses 3 important ideas in Twisted:
- The reactor
- Transports
- Protocols
The reactor
The core of Twisted is the reactor event loop. The reactor listens for network, filesystem, and timer events and dispatches them as they arrive to waiting event handlers. This is what makes Twisted event-driven: rather than blocking while waiting for expensive events like network requests to complete, or using threads to handle them, callbacks are registered with the event loop so events can be handled exactly when they are ready.
In our echo server, the reactor’s listenTCP method takes care of registering callbacks with the reactor to get notified when data is available to read from a TCP socket on port 8000. After those callbacks have been registered, we start the reactor’s event loop with reactor.run. Once running, the reactor will poll for and dispatch events forever or until reactor.stop is called.
Transports
A transport represents the connection between two endpoints communicating over a network. Transports describe connection details: for example, is this connection stream-oriented (like TCP) or datagram-oriented (like UDP)? TCP, UDP, Unix sockets, and serial ports are examples of transports.
In our echo server, TCP is the transport. Data is echoed to connected clients with transport.write.
Protocols
Protocols describe how to process network events asynchronously. Twisted maintains implementations for many popular application protocols, including HTTP, Telnet, DNS, and IMAP.
In our echo server, we create our own Echo protocol by subclassing protocol.Protocol. To echo data back to the client, we take the data received from the client and simply write it back out through the transport in Echo.dataReceived.
A common Twisted idiom is to use factories to generate instances of protocols, as we do in this example with EchoFactory.
More Twisted servers
We can use these Twisted primitives to build simple services for a variety of protocols that will look structurally very similar to the echo server.
What if we instead needed a UDP echo server? Try:
from twisted.internet import protocol, reactor
class Echo(protocol.DatagramProtocol):
def datagramReceived(self, data, (host, port)):
self.transport.write(data, (host, port))
reactor.listenUDP(8000, Echo())
reactor.run()
Or an SSL echo server? Here you go:
from twisted.internet import protocol, reactor, ssl
class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
class EchoFactory(protocol.Factory):
def buildProtocol(self, addr):
return Echo()
context = DefaultOpenSSLContextFactory("server.key", "server.crt")
reactor.listenSSL(4333, EchoFactory(), context)
reactor.run()
Twisted comes with higher-level APIs for application-layer network protocols. For example, rather than implementing the HTTP protocol ourselves, we can use twisted.web‘s APIs for serving static and dynamic content, HTTP authentication, session management, and more.
A simple web server serving static content out of a directory could be implemented like this:
from twisted.internet import reactor
from twisted.web import server, static
resource = static.File('/var/www/')
factory = server.Site(resource)
reactor.listenTCP(8000, factory)
reactor.run()
If we save this example to httpserver.py and run it with python httpserver.py, we can then visit http://localhost:8000 in a web browser to browse the directory listing for /var/www/.
Ready to dive in?
This blog post gave a brief introduction to the structure of Twisted services and a taste of what is possible with the library. For an in-depth introduction, check out Twisted Networking Essentials, 2nd Edition. In it you’ll find information on:
- An architectural overview of Twisted and event-driven programming
- Writing basic clients and servers
- More practice through a detailed look at HTTP clients and servers
- Building production-grade servers with features like logging, database access, authentication, and using threads and processes in a Twisted-safe way
- Deploying your services using the Twisted application infrastructure
- Testing your Twisted applications
- Even more practice with clients and servers for IRC, SMTP, IMAP, POP3, and SSH
By the end of the book, you’ll have all of the tools you need to build and deploy event-driven clients and servers for almost any protocol.
Jessica McKellar will present a hands-on webcast that will give an architectural overview of a Twisted project and then dive into what you need to know to build robust clients and servers for popular and custom network protocols on June 6, 2013.