Introduction of changestream-to-redis

TL;DR; Read “On Oplog Replacement in Meteor” and check out changestream-to-redis.


Hello everyone! As you may have seen, I’ve been involved in the MongoDB integration in Meteor quite heavily recently (e.g., meteor/meteor#12643, meteor/meteor#12462, meteor/meteor#12295,meteor/meteor#12116). I still plan to do more, but at the moment, my resources got rather constrained.

However! I just released a new tool called changestream-to-redis. It’s basically an automated way of publishing your cultofcoders:redis-oplog’s “outside mutations” to Redis, and thus a direct oplogtoredis replacement. It’s still in its early stages, and I plan to work on it more in the nearest future, but it already serves me locally and manages to pass an extensive E2E test suite of one of the apps.

Why a new tool? Mostly because I wanted to try something completely different – using MongoDB Change Streams instead of relying on oplog. It also has a huge performance improvement potential, as it’s now possible to work with protectAgainstRaceConditions: false.

There’s more! While working on it, I made a write up of how Meteor handles oplog, and how cultofcoders:redis-oplog actually helps. You can find it on r/programming, r/meteor, and finally, my blog.

25 Likes

Great blog post! Thanks for writing it up and the work you’ve done with change streams. As you eluded to in your last paragraph, particularly excited to see if change streams could wholesale replace oplog without the need for redis.

WOW, this is a big one.

2 Likes

Great blogpost @radekmie!!

1 Like

Thanks for working on solutions like these.

I have never used the “original” oplog nor have I experienced problems but maybe with 14 million documents my database is still too small.

Wouldn’t it be better to avoid pub/sub very possible in the first place?

pub/sub is powerful. I like using it, but it needs to be used wisely.

1 Like

Yeah, that’d be a big thing also technically (to implement). But I’m still thinking about how to do it smartly. I thought about skipping mergebox entirely, but it wouldn’t work for publications with limit or skip… Anyway, it’s an open topic.

It depends on your use case. As I wrote in a couple of places, one of the main apps we use it for is a system where the real-time part is crucial for the business. We do publish quite a lot (both count- and bytes-wise), even from large collections (millions or even tens of millions of documents).

Some collections have small documents (1KB), and some have large ones (50KB). They also differ in the update frequency (the highest one is at ~40% of the entire database). We even have to denormalize some data here and there, peaking at ~15K updates/second. All of that is still fine with real-time on a 1CPU Redis instance.

2 Likes

TL;DR; 2-4x CPU and 4-5x RAM reductions!

Some time went by, and we tested it both locally and in CI. Then we rolled it out to our pre-production, and finally production environments. Here are the results with some screenshots.

On pre-production (not much traffic, occasional peaks; e.g., migrations), we see a drop from ~1.2% CPU to ~0.3% CPU and much more stable performance. RAM-wise, drop from ~2% to 0.4%. All of that is on a 0.25CPU/0.5GB RAM AWS Fargate container.


On production (lots of traffic, regular peaks; e.g., cron jobs and migrations), we see no CPU change and significant RAM reduction (1.8% → 0.8%). But there’s a twist – we scaled down the instance by a factor of two! (It was on a 0.5CPU/1GB RAM AWS Fargate container; now it’s 0.25CPU/0.5GB RAM.) So it effectively halved the CPU usage and dropped the RAM requirements 4 times.


Remember: your mileage may vary!

11 Likes

Hi @radekmie, first of all congrats! Amazing results.

Questions:

  • Are you planning to remove this warning now from the GitHub readme? Or are there other milestones that you want to achieve first?

The project is currently in its alpha phase. There are no production loads using it yet nor any large-scale tests were conducted. I do not recommend going live with it yet.

I’m asking that because clients are not comfortable using it before this warning is removed.

  • Is there a branch of this solution running with Meteor 3?

I think we would need a branch of redis-oplog compatible with Meteor 3 and everything should work, correct?

I’m asking that because I’m already running some perf benchmarks comparing Meteor 2 and Meteor 3.

My main goal is to verify the hypothesis that Meteor 3 would be faster and lighter as we no longer need to handle Fibers.

The next step would be to also test with Redis-oplog or Redis-oplog + Changestream-to-redis-oplog :slight_smile:

Thanks!

3 Likes

Yes, definitely! I plan to add a performance section showing those results, too.

There’s nothing on changestream-to-redis that would be incompatible, but I honestly don’t know about cultofcoders:redis-oplog and Meteor 3 – didn’t get a chance yet. I can imagine it won’t be easy since Redis communication is asynchronous… But do let me know once you test it!

2 Likes

Just wanted to give a shoutout to radekmie for this jewel! We’re benchmarking this on our meteor AWS test servers and it is so efficient.

For anybody else out there also using it, do you find that the docker instance running changestream-to-redis maxes out around 30%?

3 Likes

I’m glad you like it! Here’s an update from our deployment at aleno.me:

It maxed out the CPU only ~5 times in the last 6 months. And yes, it’s still a 0.25 vCPU / 0.5 GB RAM AWS Fargate container :sweat_smile:

5 Likes

Awesome. What about your database server? have you seen any increment?

Nothing measurable really. I was afraid we’ll see an increase in RAM, but there was none.

It may be important to mention we were already using Change Streams for other reasons (MongoDB Atlas Triggers), so the data was already there anyway.

1 Like