And the results? I think they’re crazily good! Across three days, the number of fetched documents dropped from over 453 to roughly 35 million – almost 93% less. But more importantly, the number of oplog notifications dropped from over 306 to less than 1.5 million – that’s over 99.5% less!
If you have any questions, do share them here. And do let me know how much it helped your Meteor app!
Great! It’s awesome to see cultofcoders:redis-oplog, a widely used scaling tool for Meteor reactivity, gain integrations that extend its capabilities and performance. The changestream-to-redis server looks promising for projects that need high scalability.
Congrats on this work, and I love to see numbers on the latest updates.
I’m curious if you plan to add tests for changestream-to-redis. It’s easy to introduce a breaking change at some point. How do you handle that? I’m sure projects you work on have their own checks and serve as enough validation for now, but a dedicated suite would help catch regressions also for anyone that would like to help expand it.
For example, you could fork the Meteor Core and redis-oplog tests to cover built-in Redis, redis-oplog and changestream-to-redis; and ensure not breaking common behaviors on this libs, along adding other expectations.
Do you think this could be useful? Do you have a plan for it? How do you validate stability? I’m just curious about this.
Fantastic stuff as always Radek! We started using redis for our jobs queue a while back so now it seems even easier to take this step. What sort of load do you have on the actual redis server ? I’m wondering if we can just add another DB to our existing one.
Also, on the Mongo server, does it add more load ? How many connections / streams are required ? 1 per collection ?
In your docs you mention a couple of limitations
No change stream resumption. It is planned, but at the moment the program is entirely stateless.
Has this ever happened to you? So assuming we can receive an alert for this happening it’s just a case of restarting the service ?
No MongoDB error handling. As soon as the change stream fails, the program exits. It is planned, though changestream-to-redis is meant to restart as soon as it exits.
I don’t follow. Are you saying it does restart if there’s a failure or that’s something that’s planned ?
This project has always appealed to me as possibly a cool way to create an auditlog. We’ve currently implemented it in our meteor server with hooks. The benefit of doing it there is that it has all code defining the schema and can transform them into something more human readable. But I’m wondering if the best option would be to store the data here and then do the transform later on in our meteor code. This is the library we’ve been using to create the patches - I guess we’d need something similar in rust.
Currently, the entire test suite of aleno serves as the only testing ground. We have plenty of unit, integration, and end-to-end tests that are enough, at least for our needs. I plan to add a proper test suite at some point, but definitely nothing involving Meteor, but based on MongoDB and Redis only (i.e., check what messages were published given a series of database operations).
However, I’m not doing that just yet for two reasons:
We didn’t see any bugs over the past two years. Mostly thanks to the aforementioned test suite.
No bugs were reported (except for one performance-related topic, which led to the introduction of batched Redis operations).
It peaks at 4% of a single core. Memory-wise, it’s usually around 90MB with an all-time high of 140MB. And that’s not an isolated instance dedicated to changestream-to-redis – we use it for caching as well.
No measurable change, neither when we first deployed it nor when we configured namespaces.
If you won’t use FULL_DOCUMENT_COLLECTIONS, there will be exactly one connection with one change stream. Collections are filtered in a $match phase (if you’re using EXCLUDED_COLLECTIONS).
Only when we restarted it manually (e.g., due to a configuration change). It’s using less than 0.5% CPU and 7MB of RAM, so it’s not a problem to keep it stateful.
It does restart on failure, and error handling is planned. However, we saw it restart only once, when we fell behind with the oplog. It never happened since we configured batched Redis operations.
Well, you just described our History service. I won’t share the details here, but we use changestreams with pre- and post-images, and store diffs in the database. It’s not built using changestream-to-redis, though.
No, you have to read everything, including my off-topic ramblings about working at the university and emojis. Otherwise, you’re losing too much context and cannot fully experience the performance-related posts!
Yeah, I was thinking about it for some time now. Will see what it’d look like.
We started using redis for our jobs queue a while back so now it seems even easier to take this step. What sort of load do you have on the actual redis server ? I’m wondering if we can just add another DB to our existing one.