Scaling Microservices with Crossbar.io

Scaling microservices with Crossbar.io

Software Engineering Camp 2016, Tobias Oberstein, Alexander Gödde

http://goo.gl/vUUb5e

This presentation is at
http://crossbario.com/static/presentations/microservices

Minimal demo code is at
https://github.com/crossbario/crossbarexamples/tree/master/scaling-microservices

TSP demo code is at
https://github.com/crossbario/crossbarexamples/tree/master/demos/salesman

The what and why of Microservices

The most traditional model of software development is to have an application as a single big blob of code - a monolith. This seems natural - as long as ... well, there are quite a few conditions where this falls apart. There is the deployment angle: You need to re-deploy the entire monolith, which may take considerable time to start up. There is the testing angle: It's difficult to test a monolith. You probably have your own experiences here. There have been many approaches to breaking up the monoliths and allowing applications to be composed from services over the years. The current one is Microservices. Unlike previous approaches, with Microservices the concept is to have dump pipes and smart endpoints. You have a connection fabric which is fully generic and does not contain any business or domain logic. Connecting things is a light-weight process. This approach may not be perfect, but it avoids pitfalls which we know to exist based on past failed experiments.

Taming Complexity

and avoiding

Taming Complexity: getting there

Divide & Conquer
One thing, one responsibility
Decoupling

Older techs:

CORBA
DCOM
SOAP/WSDL
All of above suck BIG time
We'll see why REST/HTTP isn't really an answer and why we use WAMP

Scaling Microservices using the Traveling Salesman Problem (TSP) as an example

TSP

points representing the cities the salesman needs to visit

TSP

11 cities: 39,916,800 combinations

30 cities: 2,652,528,598,121,910,586,363,084,800,000 combinations

One Algorithm: Simulated Annealing

Scaling the TSP

The problem: there are no 1THz cores (because physics)

Part of breaking up monoliths is also the aspect of scaling. Single-core performance plateaued a long time ago. Single-machine performance is still rising, but the limits are clear. Plus you need to engineer your application for multiple threads - and you need to do so intelligently, so this is not an easy way out to begin with. Scaling is also mostly not something you need to do to your entire application. In basically any application there are hotspots. There are parts of your code which consume a lot of computing resources, and others which get called infrequently. Things that do data crunching can use scaling, while the code for an administration UI does not require more resources when the general application load increases. With a monolith, once the limits of your machine are reached, all you can do is start up another instance of the entire application. This means that you run everything multiple times - the stuff that really doesn't get called very much plus the hotspots. That's ugly overhead. You're also running multiple instances of the application - so you need to coordinate between these. What you really want to do is to scale only the hotspots - within a single instance of the application. When your application is composed of microservices, this is something that you can do.

Simulated Annealing is trivial to parallelize.

We could

use some tech specialized for parallelization of compute
use OpenMP to utilize all cores on single machine
use MPI to utilize all machines in a local cluster

or we could use Microservices

A TSP App

The app itself provides functions such as the possibility to change the parameters, a UI which displays the solution and, of course, the compute core. You can easily implement this the classic way as a monolithic application. Computing TSP solutions can be parallelized easily though, and we could really benefit from running as many compute cores at the same time as possible. THis could be across multiple cores on the same system or, today, often across multiple machines. We may also want to allow the UI to be shown on more than one machine, e.g. when multiple people want to access the best route at the same time. For scaling the compute core there are multiple existing approaches. On the same machine you could use something like OpenMP which does scaling across available cores. There are message passing interfaces (MPIs) which work across multiple machines in the same room, connected by something like Infiniband. These are only for specific circumstances and, anyway: If we structure an application as microservices, shouldn't we be able to use the microservice mechanisms to implement the scaling?

A TSP Microservices App

If we split things up into microservices, this is one possible outcome. We have several compute nodes. THere is an orchestrator to coordinate these. There are two kinds of UI - one to administrate the app and one to show the results. When looking at a microservice application, I find it natural to not think of the application and the UI as separate things. The UI components are services just like the orchestrator or the compute nodes. We'll see in a bit that the connection technology we suggest can handle both using the same mechanisms. BUt this image shows that something has been missing when we've been talking about microservices so far: We haven't talked about how they actually interact. Let's first take a look at the interactions between the orchestrator and the compute nodes.

Microservice Interactions - Calls

Microservice Interactions - Distribute Information

There is more though: Parts of the application need information about what is happening. An obvious example would be the current compute load in each component, which we want to show in the admin and user UIs. What we have here are just the interactions based on their outcomes. Let's take a look at how these interactions can be achieved. When looking at the second interaction here, we see that the distribution of information to multiple components in the application has some problems if it happens as we've drawn it. The compute component needs to know about each and every consumer of the event. The component and the consumer need to be able to establish a connection between themselves. And the component then needs to send the event to each and every consumer. This means strong coupling of components, does not work if e.g. a component is behind a NAT where we can't open ports - and is a disaster regarding scalability.

So we need

Remote Procedure Calls (RPC)
Publish & Subscribe (PubSub)

Ideally in one protocol and using one tech.

Web Application Messaging Protocol (WAMP)

WAMP (Web Application Messaging Protocol) fits that bill
WAMP is a routed protocol, requires a WAMP router
Crossbar.io is one WAMP router (ours)

open
13 languages
24 client, 12 router implementations (22/11 OSS)

The solution I want to present uses the Web Applicaiton Messaging Protocol (WAMP), which I am the main designer of. Yes, WAMP is a bad acronym, but it's too late to change that. It's too late because there are a whole lot of implementations out there. There is currently WAMP support for 12 languages through open source libraries. I and my company support Python, JavaScript and C++, with the other languages contributed by the community. There is a total of xx client implementations, and xx router implementations. Crossbar.io, our main project, is one of these routers. It is the best, and we are investing in keeping it at the top of the ecosystem. Router implementations? WAMP is a routed protocol - which is required for PubSub.

Sessions

                  
var connection = new autobahn.Connection({
         url: 'ws://127.0.0.1:9000/',
         realm: 'realm1'
      });

connection.onopen = function (session, details) {
      console.log("connected");
};

connection.onclose = function(reason, details) {
      console.log("connection closed/lost", reason, details)
}

connection.open();

Publish & Subscribe

The router distributes this as an event to all subscribers. So we get decoupling: The publisher only needs to know the identity of the router and be able to connect to it. The same goes for the subscribers. This also scales well, since the burden of messaging to multiple recipients lies with the central router, not with each component issuing the publication. Now that's one of the two patterns we saw in our demo application, and that microservice applications generally require. There are a lot of PubSub protocols and solutions out there. The other interaction was calling a procedure. With that we also don't want strong coupling and, not just because this is the topic of this talk, in some cases scaling. Do remote procedure calls need to be component-to-component and thus strongly coupled? Or can we route them as well?

                  
function onhello(args) {
   console.log("Got event:", args[0]);
}

session.subscribe('com.myapp.hello', onhello);

Publish

                  
session.publish('com.myapp.hello', ['Hello, world!']);

Routed Remote Procedure Calls

and this is passed on to the caller. SO we get decoupling just as with PubSub - which is already really nice. The caller does not need to know the identity of the provided of the procedure. It does not need to be able to connect directly to it - which is e.g. important when the provider is behind a NAT. So using this you can implement a procedure which runs in a mobile browser connected via a mobile network - and call this from anywhere in your application. It does not matter where the component is - at least at a logical level. Of course, practically speaking, connectivity considerations like bandwidth and latency play a role, as well as the other capabilities of the device a component is running on. NOt only can the procedure run anywhere, but where it runs can change between calls. THe caller only cares about the result, and if the component initially called deregisters and another registers between calls, this does not matter to it.

Register

                  
function add2(args) {
   return args[0] + args[1];
}

session.register('com.myapp.add2', add2);

Call

                  
session.call('com.myapp.add2', [2, 3]).then(
   function (result) {
      console.log("Got result:", result);
   },
   function (err) {
      console.log("Error calling procedure", err);
   }
);

Combining actions

Call a procedure from an event handler
Publish events from a registered procedure
Dynamically subcribe/unsubscribe in a procedure
Dynamically register/unregister in an event handler
...

Shared Registrations

Shared Registrations - Hot Standby

Shared Registrations - Scale-Out

Shared Registrations

                  
session.register('com.myapp.add2', add2, {
                                    invoke: 'roundrobin'
                                 });

Shared Registrations - Controlling Concurrency

Shared Registrations with Max Concurrency Limits

                  
session.register('com.myapp.add2', add2, {
                           'invoke': 'roundrobin',
                           'concurrency': 4
                        });

The TSP App

Summary

Microservices: new answer, old problem
Connection patterns: RPC + PubSub
Crossbar.io + Autobahn make this easy
Scaling and hot standby are offered by the router