Twisted at Mediamatic
Even before I got to work for Mediamatic Lab, Mediamatic was using Twisted. My friend Andy Smith used it for a bunch of projects around physical objects, usually involving some kind of RF tags. Examples include the Symbolic Table and the Friend Drinking Station. From this grew fizzjik, a Twisted based library that implements support for several kinds of RFID readers, network monitoring and access to online services like Flickr and of course anyMeta.
On the other hand, I have dabbled in Twisted for quite a while now, mostly contributing XMPP support in Twisted Words and through the playground that is known as Wokkel. But why go through all that effort, while there are a several different Python-based XMPP implementations out there? And why does Mediamatic use Twisted? Why do I believe Twisted is awesome?
First of all, we like Python. It is a great little language with
extensive library support (batteries included
), where
everything is an object. Much like in anyMeta. It is a language for
learning to program, to code small utility scripts, but also for entire
applications.
But going beyond that, building applications that interact with different network protocols and many connections all at the same time is a different story. Many approach such a challenge by using preemtive threading. Threads are hard. Really hard. And Python has the GIL, allowing the interpreter to only execute byte codes in one thread at a time.
So in comes Twisted. Twisted is a framework for building networked applications in Python, through a concept known as cooperative multitasking. It uses an event loop that hands off processing of events (like incoming data on a socket or a timer going off) to non-blocking functions. Events loops are mostly known from GUI toolkits like GTK, and so Twisted goes even beyond networking by working with such toolkits' event loops, too. As most network protocol implementations only have a synchronous interface (i.e. one that blocks), Twisted includes asynchronous implementations of a long list of network protocols. For the blocking interfaces that come from C libraries, like databases, Twisted provides a way to work with their threads, while keeping all your controlling code in the main thread. Asynchronous programming does take some getting used to, hence Twisted's name.
So how do we use Twisted? Well, a recent application is our recent RFID polling system. It allows people to use their ikTag (or any card or other object with a Mifare tag), tied to their user account on an anyMeta site, to take part in a poll by having their tag read at an RFID reader corresponding to a possible answer. The implementation involves:
Communication with one or more RFID readers (that also have output capabilities for hooking up lamps, for example).
Communication over HTTP to access anyMeta's API to store and process votes.
Network availability monitoring (can we access the network and specifically our anyMeta site?)
Power management monitoring (do we still have power?)
Device monitoring. Are RFID readers plugged in or not? Which device handle are they tied to? Also watch coming or leaving devices.
A (GTK-based) diagnostic GUI
Additionally, we also want to show polling results, so we have a browser talking to a local HTTP server and a listener for XMPP publish-subscribe notifications.
This is quite a list of tasks for something as seemingly simple as a polling stations. But wait: there can be multiple readers tied to a particular poll answer, likely physically apart, a polling question can have maybe 50 answers (depending on the type of poll, like choosing from a collection of keywords) or there could be a lot of questions at one event.
So, back to Twisted. Twisted has HTTP and XMPP protocol support (both client and server-side), can talk to serial devices (like your Arduino board) and DBus (for watching NetworkManager and device events) and provides event loop integration with GTK to also process GUI events and manipulate widgets based on events. Together with Wokkel, it powers the exchange of information in our (and your?) federating social networking sites. In Python. No threads and associated locking. In rediculously small amounts of code. That's why.
Not yet convinced? Add a Manhole to your application server, SSH into it, and get an interactive, syntax highlighted Python prompt with live objects from your application. Yes, really.