Deepstream: an Open-source Server for Building Realtime Apps

Elton, the deepstream mascot

[special]Realtime apps are getting really popular, but they're also hard to build. Wolfram Hempel introduces deepstream, an open-source server he co-founded to make data-sync, request-response and publish-subscribe a whole lot easier.[/special]

The Rise of Realtime Apps

Realtime is eating the world! Or at least it’s taking bigger and bigger bites. Whether it's collaborative editing in Google Docs, chatting via Facebook messenger, financial trading on the move, IoT controls, live dashboards or multiplayer gaming --- users are increasingly expecting to see changes happen as they happen.

Even traditionally static sites like social networks or forums are starting to abandon the refresh button and instead stream updates directly into your feed.

But as popular as realtime apps are, they're also hard to build. While it's possible for smaller projects or POCs to introduce realtime features just by adding a pinch of Socket.io, large-scale use cases require a fundamentally different architecture. Concepts like concurrent connections, failover, streaming data-consistency, persistence, encryption and permissioning all have to be woven into the fabric that powers this new generation of apps.

One industry where I learned this only too well is investment banking. The servers that power the myriad of flashing screens on the world’s trading floors are monolithic beasts, complicated and breathtakingly expensive. But they are fast. Very fast. And they’ve got something else right: they use a concept called “data-sync”.

Realtime Concepts

If you’ve already built a realtime app, chances are you’ve used a pattern called “publish-subscribe” or “pub-sub” for short: subscribers listen for events on a channel and others publish these events. It’s an efficient mechanism for many-to-many communication that’s supported by a wide range of technologies and services such as Socket.io or SocketCluster on the open source side, or Pusher, PubNub or Ably in the PaaS space.

But there's one crucial thing that pub-sub can’t provide: state. Pretty much every app has some state --- data that needs to be created, read, updated and deleted, but pub-sub only delivers one-off messages that vanish immediately. A common workaround is the use of events as update notifications which in turn prompt the client to retrieve the latest state via traditional request-response. But that’s complicated, prone to inconsistencies and, most importantly, slow.

This has led to the increasing move towards data-sync, an approach that combines statefulness with realtime updates. Data is persistent as well as kept in sync between connected clients and backend processes.

Technologies that support this are much rarer. On the open-source side there used to be the now discontinued horizon.io; in the PaaS space there’s Google’s Firebase.

To fill this gap, we've started deepstream.io. Our aim was to create an open-source server with the same performance and versatility that financial trading or multiplayer gaming systems deliver, but in an open and extendable way that makes it easy to use for any kind of app.

Deepstream.io

Deepstream is a new type of server that handles realtime data at scale. Its installed similar to a database or an HTTP server. End users and backend services connect to it via lightweight SDKs that come in a range of different programming languages, such as JS/Node, Java/Android, or Swift/ObjC.

It provides data-sync as well as pub-sub and classic request-response and caters for a wide range of functional requirements such as failover, permissioning, encryption, consistency and conflict resolution.

Deepstream is designed to thrive in open-source ecosystems, and comes with a range of connectors for popular databases, caches or message busses.

But most importantly: it's scalable, reliable and very fast.

Continue reading %Deepstream: an Open-source Server for Building Realtime Apps%

by Wolfram Hempel via SitePoint