Event sourcing with Apollo


#1

Hi,

I’m working on rewriting Wekan with the goal of having a modular architecture that will allow users to install a variety of extensions to extend their board features (read more on that). I decided to use event sourcing as my server-side data handling architecture, as it provides a lot of valuable benefits for a modular application—eg, you don’t lose the previous data while updating a field, it helps solving the modeling problem (normalization vs denormalization), and many others. If you aren’t familiar with event sourcing yet, I recommended watching this introduction:

So I wonder how I could use this pattern with Apollo. There is already some research on Event Sourcing with GraphQL, for instance this gist and this working application. These examples rely on a not yet standardized subscription operation that is being discussed in the GraphQL community and that Apollo will probably want to use for reactive queries.

So Apollo crew, I’m curious. Have you already thought about using this architecture with Apollo? Do you identify some specific challenges in integrating the two data models?


#2

another good resource:

https://ookami86.github.io/event-sourcing-in-practice/


#3

Yes, event sourcing is definitely a model we’re most likely going to support, but at the moment we’re focusing on getting the foundation right, so we haven’t spent too much time thinking about it yet.

One of the main challenges I see is that not all clients are created equal, and many clients will prefer simplicity. When you have an asynchronous model on one side and a synchronous one on the other, a lot of interesting stuff has to happen at the middle layer, and that’s exactly where Apollo would be.


#4

The more I read about GraphQl/Apollo and understanding Event Sourcing , the more I am convinced that GraphQL is a good candidate to implement Event Sourcing/CQRS pattern. Just the fact that GraphsQL allows two different query types - Queries and Mutation is a conceptual direct map to the basics of the Event Sourcing pattern of separating the reads and writes, gives a good foundation to explore this pattern, alongside other advantages.

I am predicting lot more projects would be using Event Sourcing with GraphQL as inherent complexity of the implementation would be reduced by the kind of advantages of GraphQL provides and there is a symbiotic relationship between the two.

Below are some more good resources for Event Sourcing/CQRS pattern, just in case any one is interested


http://docs.geteventstore.com/introduction/event-sourcing-basics/
http://www.tomseldon.co.uk/blog/event-sourcing-in-javascript/


#5

Can someone post a 1-paragraph explanation of event sourcing? Every resource i’ve found is like some 20 page thing, so it’s hard to bother to understand it.


#6

Event sourcing is basically creating a log of all your method calls/mutations (operation and arguments), so you can run/re-run them at any time (provided you have the correct initial state).

Event sourcing can them be combined with this architecture called CQRS - Command Query Responsibility Segregation.

Your mutations are the Command part. The Query part is the subscription, that should take the initial state and apply all the mutations on that initial state and return the result.


#7

OK, so given that explanation, is it fair to say that GraphQL already follows this pattern, so there is nothing to be done? Or is this approach incompatible with traditional databases entirely?


#8

I think it is fair to say that. Today people implement this over traditional DBMSs and app servers. There is a lot to worry about to make this work, but I believe is work to be done at the application level.


#9

The simplest explanation I can provide is Event Sourcing is a way of persisting the application’s state by storing the history of events that determines the current state of the application. So it is different than the normal ORM based CRUD apps in that it is like a log of stream of events where you just append to the existing state.

The simplest analogy I can think in real life modeling scenario is: it is like a Bank Statement, where you have this series of transactions or events for the month and then you get the current balance which is the state. The advantage here is since you have all data available in the event history you can create all kinds of models based on the event stream and create state out of this at any given point of time or may be project in to the future.

CQRS is used in conjunction with Event Sourcing, but can be used separately ( I have no idea why or how :slight_smile: ) . Essentially as @okada mentioned it separates the queries with the mutations. It essentially has two stores one for reading and other for writing… [I am still trying to understand this concept :slight_smile:

Regarding the similarity of GraphQL, this link provides better explanation than I could ever give.

[I am still wrapping my head around the concept :slight_smile:


#10

As I know that you are already familiar with Redux, you can think of Event Sourcing (ES) as Redux patterns and principles for the server-side data store—though from a chronological point of view it’s fairer to call Redux an imitation of ES for the client.

As @nata_goddanti pointed out, at the core of an ES architecture is the history of all events, stored in a queue, which is just to say that the only API we can use to mutate the app state is to append a new event. Maybe a difference with the Flux/Redux mental model is that this queue of events is the actual state of the application, it is the “single-source of truth”.

But a queue isn’t an efficient data structure for extracting informations from it, as if you want to know the current pseudonym of user whose id is 123, you basically have to process the entire queue and listen for all createUser(id:123, pseudo:'max') and renameUser(userId:123, newPseudo:'max2') events to figures that out. To make ES model suitable for reads, we create “materialized views” which are classical store of data (that could be stored in MongoDB or on any other data store) that are build by listing to some events (“actions” in the Redux world) and updating their store accordingly. In my example the renameUser event would probably fires a Users.update(123, {$set: {pseudo: 'max2'}}); update. We then read directly from this data view, but continue to write our events in the events queue.

This architecture has advantages over the traditional ˘RPC that mutate the DB” one. For one, you never loose data, previous changes are still present in your event history and so if something goes wrong (a cracker that changes your users password for instance) you are able to recover the old data. It also solves issues related to data modeling, like how much denormalization or duplication is desirable. Basically with ES you can duplicate the data as much as you want and put it in whatever structure, normalized or not, that you want because basically you are only creating a cache that doesn’t change the write model (“unidirectional data flow”). This also means that if you change your read model you don’t need to write a clever migration query without bugs, but can simply re-build it by re-running the entire stream of events and be sure to have a final state that is consistent with your mutations history. ES also helps the implementation of features like real-time updates, horizontal scaling, conflicts resolution, incremental backups, or multi-actors cooperation.

ES is indeed compatible with GraphQL, as GQL isn’t really concerned about where you read your data from and what you do on a write. But I opened this discussions to see if Apollo as the “modern data stack” could help with the ES architecture—or at least not prevent it. Especially if Apollo is evolving toward a set of composable tools to reach the needs of mosts, it woud be good to keep the ES model in mind.

And yes, sorry @sashko I failed to limit my prose to a single paragraph :slight_smile:


#11

Curious to see what we could do here! Since we’re using something similar on the client already (redux) perhaps there is something we could do to make that work together nicely with ES on the server as well :]


#12

Hi @mquandalle,

Did you end up using ES with Apollo?
You had the exact same thought 2 years ago, as I have now.
How did it go? :slight_smile:

/Rasmus


#13

I fail to see how ES and Apollo are even related. I see ES as a design pattern / datastore itself, and Apollo as the acces layer, or proxy between datastore and client.

The way I see it is basically this: https://launchpad.graphql.com/mj4501j099

Am I not understanding Event Sourcing, or do you guys think to difficult?

Of course, the ES in the launchpad above is simplified. There is no persistent storage layer for the queue, and no materialized view. But I think it shows the principle just fine.