However! I just released a new tool called changestream-to-redis. It’s basically an automated way of publishing your cultofcoders:redis-oplog’s “outside mutations” to Redis, and thus a direct oplogtoredis replacement. It’s still in its early stages, and I plan to work on it more in the nearest future, but it already serves me locally and manages to pass an extensive E2E test suite of one of the apps.
Why a new tool? Mostly because I wanted to try something completely different – using MongoDB Change Streams instead of relying on oplog. It also has a huge performance improvement potential, as it’s now possible to work with protectAgainstRaceConditions: false.
There’s more! While working on it, I made a write up of how Meteor handles oplog, and how cultofcoders:redis-oplog actually helps. You can find it on r/programming, r/meteor, and finally, my blog.
Great blog post! Thanks for writing it up and the work you’ve done with change streams. As you eluded to in your last paragraph, particularly excited to see if change streams could wholesale replace oplog without the need for redis.
Yeah, that’d be a big thing also technically (to implement). But I’m still thinking about how to do it smartly. I thought about skipping mergebox entirely, but it wouldn’t work for publications with limit or skip… Anyway, it’s an open topic.
It depends on your use case. As I wrote in a couple of places, one of the main apps we use it for is a system where the real-time part is crucial for the business. We do publish quite a lot (both count- and bytes-wise), even from large collections (millions or even tens of millions of documents).
Some collections have small documents (1KB), and some have large ones (50KB). They also differ in the update frequency (the highest one is at ~40% of the entire database). We even have to denormalize some data here and there, peaking at ~15K updates/second. All of that is still fine with real-time on a 1CPU Redis instance.
Some time went by, and we tested it both locally and in CI. Then we rolled it out to our pre-production, and finally production environments. Here are the results with some screenshots.
On pre-production (not much traffic, occasional peaks; e.g., migrations), we see a drop from ~1.2% CPU to ~0.3% CPU and much more stable performance. RAM-wise, drop from ~2% to 0.4%. All of that is on a 0.25CPU/0.5GB RAM AWS Fargate container.
On production (lots of traffic, regular peaks; e.g., cron jobs and migrations), we see no CPU change and significant RAM reduction (1.8% → 0.8%). But there’s a twist – we scaled down the instance by a factor of two! (It was on a 0.5CPU/1GB RAM AWS Fargate container; now it’s 0.25CPU/0.5GB RAM.)So it effectively halved the CPU usage and dropped the RAM requirements 4 times.
Hi @radekmie, first of all congrats! Amazing results.
Questions:
Are you planning to remove this warning now from the GitHub readme? Or are there other milestones that you want to achieve first?
The project is currently in its alpha phase. There are no production loads using it yet nor any large-scale tests were conducted. I do not recommend going live with it yet.
I’m asking that because clients are not comfortable using it before this warning is removed.
Is there a branch of this solution running with Meteor 3?
I think we would need a branch of redis-oplog compatible with Meteor 3 and everything should work, correct?
I’m asking that because I’m already running some perf benchmarks comparing Meteor 2 and Meteor 3.
My main goal is to verify the hypothesis that Meteor 3 would be faster and lighter as we no longer need to handle Fibers.
The next step would be to also test with Redis-oplog or Redis-oplog + Changestream-to-redis-oplog
Yes, definitely! I plan to add a performance section showing those results, too.
There’s nothing on changestream-to-redis that would be incompatible, but I honestly don’t know about cultofcoders:redis-oplog and Meteor 3 – didn’t get a chance yet. I can imagine it won’t be easy since Redis communication is asynchronous… But do let me know once you test it!
