I have been following Meteor for years, and I always had my reservations about MongoDB oplog tailing.
As I understand it, with the release of Meteor 3.5, oplog tailing has been replaced by change streams.
I have always thought Meteor would benefit from moving oplog tailing to Redis, and after seeing the redis-oplog package, I assumed the Meteor team would see the benefit in using Redis for change tracking and maybe integrate redis-oplog officially to Meteor. (Although I understand the caveats of having limited resources, I have the same bottleneck with the projects I’m managing.)
So my first question is, does the move to MongoDB change streams break the easy integration with redis-oplog?
I also read somewhere that redis-oplog was abandoned, but since I am not using Meteor every day (I really wish I could), I am really curious how much data users of Meteor are managing in their applications, and the resource amount they employ. (I know you can’t exactly quantify the phrase “how much data”, but it should give me some idea about resource requirements.)
This is a good question. I would assume the underlying functionality of redis-oplog doesn’t automatically work with how Meteor and change streams works. And that enabling change streams deprecates the need for redis-oplog. But I wouldn’t mind some official Meteor team insight on this.
We are currently running our production app on Meteor 3.3.2 using the original cultofcoders:redis-oplog@3.0.1 So it is still running good in production.
From looking at this redis oplog is generally better than changestreams which IMO makes sense because redis is minimizing the amount of information being thrown around and a the delegation of information is offloaded onto a different server that has no other purpose.
From my testing redis continues to work just fine in 3.5 and IMO if you have already started using it you should continue using redis
Um, I see several oplog runs but not redis-oplog. Are you sure the oplog runs are using redis-oplog and not just regular Meteor MongoDB oplog monitoring?
No, it does not. For example, if you remove a lot of documents within your collection (running a cleaning job), the new change streams implementation will still crash your Meteor instance, because you are not able to decide which commands should be visible to Meteor and which not.
Ah you were correct. Anyways I have published my production app using redis-oplog with 3.5 with essentially no ill effects and have seen a lot less garbage collection.
The primary motivation for migrating away from Oplog isn’t solely performance, but rather maintainability and the need to modernize our internal APIs. Oplog is an older, internal MongoDB API (implemented over 10 years ago) that lacks comprehensive documentation, which has forced us to rely heavily on reverse engineering in the past. Furthermore, MongoDB could potentially alter or deprecate it in future versions without significant warning. In fact, it is already unsupported on Atlas Shared and Atlas Serverless.
Change Streams are MongoDB’s official and recommended API for real-time reactivity. The performance improvements are a welcome bonus, especially in scenarios where third-party applications perform heavy database I/O, which previously pressured the Meteor app to filter out all non-related data from the reactivity system.
Regarding the decision not to include redis-oplog: our goal was to avoid introducing a new dependency. Meteor is already tightly coupled with MongoDB, IMO it’s a significant disadvantage nowadays. We do have long-term plans to make the framework more database-agnostic (you can check out a POC right here), but our immediate priority is to adopt modern, officially supported technologies first.
At the same time, the vast majority of large Meteor deployments I worked on or audited were using it. It would be great to see whether the change is safe or may wake you up with an outage. (Especially when combined with changestream-to-redis, which is already based on change streams. And no, it’s not a self-promotion – I got asked this question a few times already and had to suggest a test rollout.)