Is Crossbar.io the Future of Python Web Apps?

Crossbar.io - next-gen app server with big potential. An independent review by Sam & Max.

The Right Thing - Badly Explained¶

A guest post by Sam from Sam & Max. Original (French) version and Reddit discussion. More related posts:

• Small demo of a pragmatic use of WAMP in Python
• Free Your Code - Backends in the Browser

I have been following Crossbar.io for some time now, and I'm very, very surprised to not hear more about it in the Python community.

Well, in fact, I'm not totally surprised.

On the one hand, it's a tech that, in my opinion, represents what Python should aim for in order to finally be BFFL (Best Friends For Life) with Go/NodeJS and offer, at last, a killer feature in the word of complex server applications.

But on the other hand, well, their home page explains what it's for with this blob :

Crossbar.io is an application router which implements the Web Application Messaging Protocol (WAMP). WAMP provides asynchronous Remote Procedure Calls and Publish & Subscribe (with WebSocket being one transport option) and allows to connect application components in distributed systems

Yes, sir, yes, but, err... In real life, I mean, what the hell can I do with it ?

It's always the same problem with smart people: they make uber cool stuff, and nobody got a clue of its usefulness because they can't manage to explain it.

Since I'm a bit dumb myself, I'm gonna take this opportunity to share the love.

Web Application Message Protocol (WAMP)¶

Crossbar.io allows several parts of an application to communicate through it, using the WAMP protocol. Of course, a WAMP client can talk to the server and vice-versa, just like a HTTP client can talk to Apache/Nginx. But more than that, WAMP clients can talk transparently and easily to each other, the way an AMQP client can talk to others through a RabbitMQ server.

Still, this doesn't help you much if you don't know what WAMP is. Putting the cart before selling the bear's skin, or something like that.

WAMP is a standardized protocol built with real time message exchange in mind. It's not tied to Python, even though Crossbar.io implements it in Python. You can speak WAMP in any language.

It works mainly on top of Websocket, so you can use it directly in the browser, in Firefox, Chrome, Opera, Safari, and even IE10, using a Javascript library. Which is just a big wrapper around the Websocket API standardizing data transmission. There is nothing magical behind it, no complicated format, no binary data, just plain vanilla websocket with JSON formatted data using certain conventions. In that sense it compares to SockJS and (RIP) socket.io.

But unlike SockJS, it is not limited to use in the browser. There are libraries to use it in Python, C++ or NodeJS server codes, in Android) apps, and even directly in Nginx guts or from an Oracle database (in SQL) with some routers.

You must understand this, it's very important, it means you can send and receive arbitrary data in real time between all these systems, without headache, in a congruent and transparent way.

In summary, WAMP is like, but better than :

• HTTP requests, because it's push, asynchronous and real time;
• websocket using SockJS, cause it's a standard, and works on and between server side components/services, even using different languages.
• AMQP messages, since it works in the browser, and is easy to setup.

Used intelligently, Crossbar.io can let Python join in the innovative real time frameworks playground with others such as MeteorJs, and potentially, be even better.

Because WAMP allows you to do 2 things, easily, and well.

1 - PUB/SUB¶

It's saying "call this function when this event arrives". It's like Django signals or QT slots, but it works across the network. This is what is often currently done with Redis nowaday. Using WAMP and Javascript, it looks like this :

// connecting to the WAMP router (E.G: crossbar.io)
ab.connect("ws://localhost:9000", function(session) {
    // say you are interedted in this event
    session.subscribe('the_event_you_like', function(topic, evt){
        // do stuff with data sent with the event
        // every time the event is sent
        // such as updating the DOM
    });
});

   

Somewhere else, in an other part of the file, or even in another browser :

ab.connect("ws://localhost:9000", function(session) {
    // create and event and attach data to it
    session.publish('the_event_you_like', ['some data']);
});

   

And yes, that's it. You connect to Crossbar.io, and you chat. The subscribed callback will be called with the published data. Even if there are 1000 miles between them. Even if code A is in a browser and B is on another one, or a NodeJS server, or and Android app.

What's scary at the beginning is that you got TOO MUCH flexibility :

• What kind of event should I expect / trigger ?
• What kind of data should I send / received ?
• Is it fast ? Is it light ?

But in fact it's super easy: an event is just an action from your app, such as adding an element (a post, a comment, a user), deleting it, modifying it. In the end, it's good ol' CRUD, but in real time, and with push instead of pull. You choose a name describing your action, you attach data to it, and voila, you got an event all the subscribers can receive.

With a bonus though: it works on the server too! Your Python code receives "add a comment" as an event? It can add the comment in database, send a message to a cache service, or another web site in NodeJS to update it, send back and event to update all the web pages and the Android app, etc.

You can send any kind of data that is serializable to JSON. Anything you would send using HTTP, really. Deeply nested and complex things like geographical data - or very simple ones like notifications.

With Pub/Sub, WAMP replaces everything you would normally do with AJAX calls in the browser, and stuff you would do with messages queues on the server. What's even more powerful is that it unites the two words.

And even if it doesnt match ZeroMQ performance (which does not have a central server), it's still very fast and light.

2 - RPC¶

Calling a function located somewhere else than in its own code. It's definitely NIH (if you got painful memories of CORBA or SOAP, raise your hand), and it's very handy.

Let's keep it simple and carry on with Javascript examples, but remember you can do the same in C++ or Python:

ab.connect("ws://localhost:9000", function(session) {
   function a_function(a, b) {
      return a + b;
   }
   // you declare that this function is callable from outside
   session.register('a_function', a_function);
});

   

From the caller side :

ab.connect("ws://localhost:9000", function(session) {
    // the call remotly the function, get back a promise
    // which will let you work on the result
   session.call('une_fonction', 2, 3).then(
      function (res) {
         console.log(res);
      }
   );
});
   

Much like PUB/SUB, people have a hard time to grasp the usefulness, again because of the lot of freedom it brings. Imagine your project is now divided in a lots of small services, that run independently :

• One web service.
• One auth service.
• One API service.
• One long running task service.
• One monitoring and tech admin service.

All these services can communicate with each other using RPC, but don't need to actually run in the same process. You can (finally!), use all your server cores, and even move them to different servers if you feel like it.

Even better, having a blocking service doesn't lead to the whole system getting stuck. One major issue with Python asynchronous systems is that many libraries are still blocking (typically ORMs). With this kind of architecture, you can create a service doing only blocking calls, and let other non blocking services just be, and call the blocking one using RPC asynchronously. While the bad blocking service is stuck, the rest of the system can process other requests.

Crossbar.io, more than a WAMP router¶

The vision of the Crossbar.io developers is to allow you to build systems with composable services communicating with each other instead of some big central process. So they didn't stop at routing.

Crossbar.io is also a process manager, much like a supervisor, or more likely, circus (Tarek, stop working on that stuff, come here!) with its ZeroMQ communication stack.

You can configure it with a simple JSON file, and define Python classes that will be run in a separate process and be able to discuss using WAMP

{
   "processes": [
      { // first process
         "type": "worker",
         "modules": [
            {
               a_worker.Class
            },
            {
               an_other_worker.Class
            }
         ]
      },
      {  // second process
         "type": "worker",
         "modules": [
            {
               an_another_worker_in_another_process.Class
            }
         ]
      }
   ]
}
   

But if that's not enough, you can also run external non-Python programs, and Crossbar.io will happily handle their life cycle :

 {
    "processes": [
       {
          "type": "guest",
          "executable": "/usr/bin/node",
          "arguments": ["your_script.js"],
          "stdout": "log"
       }
    ]
 }
   

So now you have 3 powerful tools to build a decoupled, scalable, multi-core architecture (bye bye GIL!), that can compensate for blocking libraries:

• A flexible, simple protocol allowing everybody to talk to each other (WAMP).
• An API to react to change (PUB/SUB) or request an action (RPC).
• A software to manage this communication and your system components life cycle.

Concrete use case¶

WAMP is the kind of tech that does NOT allow you to do something you couldn't do before. It's not new stuff.

However, WAMP DOES allow you to do it better, and more easily.

Take the work flow of a user connecting to a forum. He goes to a form, posts his credentials, reloads the page, and he is signed in. If other users reload the page, they will see one more person is online.

Now, if you want to make that more dynamic, you can use AJAX, and if you wish to have near real time updates, you'll need to set a timeout to make regular AJAX calls or even go for long polling. It works, but it's really a hack.

Some modern sites use websockets, and asynchronous servers based on Javascript/Erlang/Go, and a PUB/SUB router on top of it like Redis. It's faster, easier, and the app is very reactive. But it's a heterogeneous system, there is nothing standard about it. Plus if you want to send messages between components (and God forbid, between components using different languages), it will require even more work.

WAMP unifies all that. A bit of RPC to sign in :

Then, a bit of PUB/SUB to notify everybody something just happened :

Or course, you COULD do that with an existing stack. It's just not as handy.

Moreover, Crossbar.io promotes separation of concerns, with a service that deals only with authentication, without needing a kraken-lookalike system to do so. If tomorrow your auth needs to have its own core/server/VM for performance or security reasons, it's possible, and easy.

Crossbar.io and WAMP, are in that sense already the best friends of our lovely new toys such as Docker.io (service composition) or AngularJS (reactive data updates).

Now, here is the catch¶

Because, of course, there is alway a catch in our sadistic job.

And it's the youth of the project. And the fact that there is only a small team working on it right now. They need help.

It's quite stable, the code works, the sources are clean.

But the doc, or dear Zalgo, the doc. The examples aren't up to date, there are two versions of the protocol which are at odds with each other, so that you often don't know anymore which part belongs to which.

And just like any young project, it lacks of out of the box definitive solution this-is-how-you-do-it-I-swear ways to tackle important problems like authentication.

But the worst is the API. Especially the Python API, which is based on Twisted, and exposed with almost no polish. If you don't know Twisted, it's a bit like Zope: powerful, robust - and the API will make you want to kill yourself.

Brace yourself, the Python 'hello world' is comming:

from twisted.python import log
from twisted.internet.defer import inlineCallbacks

# autobahn is the de facto Python lib to build
# WAMP clients
from autobahn.twisted.wamp import ApplicationSession
from autobahn.twisted.wamp import ApplicationRunner

class ListenForEvent(ApplicationSession):

    def __init__(self, config):
        ApplicationSession.__init__(self)
        self.config = config

    def onConnect(self):
        self.join(self.config.realm)

    @inlineCallbacks
    def onJoin(self, details):
        callback = lambda x: log.msg("Received event %s" % x)
        yield self.subscribe(callback, 'un_evenement')

# Python doesn't have a default event loop, so
# we need to start one
if __name__ == '__main__':
   runner = ApplicationRunner(endpoint="tcp:127.0.0.1:8080",
                              url="ws://localhost:8080/ws",
                              realm="realm1")
   runner.run(ListenForEvent)

   

This is why I lured you with the JS examples and not Python, despite the fact that I'm a Python expert and deeply in love with it.

Here is what you should be able to do if for a simple project:

from autobahn.app import App

app = App(url="ws://localhost:8080/ws")

@event("an_event")
def handle_event(details):

    app.log("Received event %s" % x)

@register()
def rpc_exposed_function(a, b):
    return a + b

if __name__ == '__main__':
   app.run()
   

Which is something we talked on the mailling list, and it's now happening. But it needs so much more work. It needs to be battle tested. It needs to be sent to bite the dirt, and back.

It needs you.

In the end¶

The Python community can be conservative and, just like WSGI became the champion because Django adopted it and pushed the need for it, an asynchronous standard will emerge only if a killer app proves the need for it.

For now we have none, because innovation is stale in the Web Python community. Partly because we love our current tools, partly because we can hack around it (async flask + gevent, celery) but also because some people are just leaving Python for Go or NodeJS. The Python 3 transition didn't help either.

Right now, we improve, but don't create that much. It's a good thing for stability, but evolution sometimes need a bit of disruption.

My suggestion about the flask-style API is not innocent. I strongly believe there is a need for the Python community to have an app providing a full stack for asynchronous web app programming in an elegant and powerful fashion. Powerful is provided by Twisted and Tornado, but they're not elegant, and certainly not sexy or pleasant.

In the end, it will be possible for some new (or old :)) Jacob Kaplan-Moss to build a framework including a flask-style web framework using cyclone and make it interact with the flask-style Autobahn. Adding some tools in the mix like:

• a live settings service,
• a dummy key/val store for dev,
• some static file tools,
• a log/stat component,
• a task queue manager,
• a search engine (like woosh) and
• modern libs (path.py, arrow, docopt, etc)

With this we would have a pure Python framework which tackles all the common tasks of today's programming, yet with a very, very simple API.

This alone could motivate people to finally build new gen ORM, make peace with asynchronous APIs and NoSQL systems in Python, and invent more.

This would be a killer app that will make people want to code stuff with it. Something the Python community lacks. Nothing evolved much since Django came out. Tornado is no fun. Twisted can do everything, but do you want to do anything with it? Ruby is dying because people are switching from Rails to NodeJS.

Python is lucky to have a more robust backing because it's not just Web oriented (plenty of innovative stuff happens in Python, just not on the Web), and because the current tools are awesome, but we do need innovation.

That's why I believe in Crossbar.io. I believe it is the first step in the creation of the work tools of tomorrow. And since WAMP works in NodeJS as well, it will make the 2 worlds merge, and that can be wonderful.

About the author

Sam is a developer mainly working in Python and JavaScript. Contact him at Sam & Max.

License

This blog post is licensed under Creative Commons CC-BY-SA.

reddit thread

There's a long, interesting reddit thread regarding this post.