However! I just released a new tool called changestream-to-redis. It’s basically an automated way of publishing your cultofcoders:redis-oplog’s “outside mutations” to Redis, and thus a direct oplogtoredis replacement. It’s still in its early stages, and I plan to work on it more in the nearest future, but it already serves me locally and manages to pass an extensive E2E test suite of one of the apps.
Great blog post! Thanks for writing it up and the work you’ve done with change streams. As you eluded to in your last paragraph, particularly excited to see if change streams could wholesale replace oplog without the need for redis.
Yeah, that’d be a big thing also technically (to implement). But I’m still thinking about how to do it smartly. I thought about skipping mergebox entirely, but it wouldn’t work for publications with limit or skip… Anyway, it’s an open topic.
It depends on your use case. As I wrote in a couple of places, one of the main apps we use it for is a system where the real-time part is crucial for the business. We do publish quite a lot (both count- and bytes-wise), even from large collections (millions or even tens of millions of documents).
Some collections have small documents (1KB), and some have large ones (50KB). They also differ in the update frequency (the highest one is at ~40% of the entire database). We even have to denormalize some data here and there, peaking at ~15K updates/second. All of that is still fine with real-time on a 1CPU Redis instance.
Some time went by, and we tested it both locally and in CI. Then we rolled it out to our pre-production, and finally production environments. Here are the results with some screenshots.
On pre-production (not much traffic, occasional peaks; e.g., migrations), we see a drop from ~1.2% CPU to ~0.3% CPU and much more stable performance. RAM-wise, drop from ~2% to 0.4%. All of that is on a 0.25CPU/0.5GB RAM AWS Fargate container.
On production (lots of traffic, regular peaks; e.g., cron jobs and migrations), we see no CPU change and significant RAM reduction (1.8% → 0.8%). But there’s a twist – we scaled down the instance by a factor of two! (It was on a 0.5CPU/1GB RAM AWS Fargate container; now it’s 0.25CPU/0.5GB RAM.)So it effectively halved the CPU usage and dropped the RAM requirements 4 times.
Yes, definitely! I plan to add a performance section showing those results, too.
There’s nothing on changestream-to-redis that would be incompatible, but I honestly don’t know about cultofcoders:redis-oplog and Meteor 3 – didn’t get a chance yet. I can imagine it won’t be easy since Redis communication is asynchronous… But do let me know once you test it!