Meteor Scaling - Redis Oplog [Status: Prod ready]


@alawi interesting idea, how will it benefit the community ?

@minhna I still don’t understand how it can have any impact on it, I’ll investigate soon.


The thinking was if it’s possible to position Redis Oplog as a generic adapter for Meteor’s pub/sub API allowing for different scaling implementation strategies one of which could be Emitter. Thus a developer could keep the same code base but just switch the initialization parameters.

I think what Emitter brings to the table is an open source docker based solution with a cloud offering and IoT support specifically tailored for pub/sub scaling. Also, Emitter has an interesting business model, it demonstrates the possibly of monetizing from the pub/sub scaling challenge, a business model that MDG or others within the meteor ecosystem could adopt.

I’m by no means familiar or affiliated with Emitter, and there are other cloud offerings such as google pub/sub and kafka on which the same arguments can be applied, just sharing some thoughts.


@alawi Have you read through the redis-oplog documentation? It already tracks very close to emitter’s proposed benefits… it has channels, a middle-man “always on” server (Redis), etc.

There’s an impressive amount of granularity and control in redis-oplog.


Just wondering, doesn’t Vent give you the same benefits? You can publish and subscribe to Redis messages without using the Mongo database. We currently using it as a “signaling server” for WebRTC messages.


Yes Vent gives you awesome granular control. However, just an FYI, you currently cannot use Vent without also using redis-oplog for all your publications (unless you disable oplog and poll all of them). So you can’t mix Vent with traditional Mongo oplog tailing. I had a use-case for this and at the moment, it can’t be done. So Vent is sort of a “bonus” when you add redis-oplog and use publishWithRedis. Issue filed here.

Probably not a big deal for most use-cases as if you use Vent with Redis you probably will use and benefit from publishWithRedis too.


For those interested in Meteor oplog scaling - the documentation for MongoDB’s Change Streams is finally live:


Please keep me honest here. My understanding is that you need setup and point to a Redis Server for the Redis-oplog to work, so far so good. But my question is why stopping at Redis? can’t the solution be generalized to act as an adapter layer with multiple potential drivers? Also can we right now point to a cloud infrastructure the does the horizontal scaling as Emitter or other pub/sub solutions do without changing the app code?

With regards Meteor Oplog, I think the high coupling of the Meteor’s pub/sub API with MongoDB did meteor a disadvantage. In my opinion, a generalized, modular, pluggable and extendable pub/sub layer at the app level is the right approach, and Redis-oplog is a step in that direction. That layer in addition to giving developers the ability to switch based on scale, it could also allow the creation of a third-party commercial hosting for Meteor’s pub/sub with horizontal scaling, stats, etc something similar to what Kadira did for monitoring.


@alawi you are right, you could easily abstract the pubsub nature of redis-oplog


@alawi Agreed as well. To abstract the pub/sub server is good. Just an FYI, I’ve done some tests on my app with 5,000+ simultaneous users all performing transactions in my app using redis-oplog and Redis’ performance graphs on barely bump at all… like 1%. So that’s why I questioned above. Is pub/sub performance so demanding we need to worry about horizontal scaling just for this component? Maybe. Not in my tests for now at least. Not a bad idea though. Abstraction is always good.

@msavin Wasn’t is discussed above that MongoDB’s Change Streams don’t offer the flexibility to solve the problem? Hence viva redis-oplog?


It depends on how you define the problem.

redis-oplog helps you scale oplog further while keeping your Meteor application and practices the same.

ChangeStreams offers a completely new approach.


Wow that is awesome. once agains thanks @diaconutheodor and team for the great work on this package, why this not on Meteor’s front page is beyond me!