I’ll be trying it in my app sometime within the next month or so when my schedule clears up. This is a really exciting alternative to oplog as well as Apollo, thanks!
1 Like
Going to try it out in the next few days. I’ve been following this thread closely and it’s exciting.
Glad you see it that way. I want bugs!
I’ll be trying it in my app sometime within the next month or so when my schedule clears up. This is a really exciting alternative to oplog as well as Apollo, thanks!
Not an alternative to Apollo, but yes it can make Meteor apps super-performant. Thanks!
1 Like
So I guess if I am using the Mongo rawCollection batch methods that I can’t really take advantage of this package since it seems to depend on extending the Meteor Mongo collection interface.
What would you suggest in this case to get batch level DB performance but to take advantage of the redis oplog?
@hexsprite
You have control over reactivity, you can take advantage of it. It does not depend on Mongo.Collection interface, it extends it, check here.
You can even use SyntheticMutation to do what you need to do. If you have idea for a feature to aid you in your task, add it as an issue.
I see. so if I wanted to add support for batch operations I could wrap my rawCollection access with something that would also issue the appropriate INSERT or UPDATE events.
And to get the most performance out of this would I basically have to convert all of the existing publish calls with publishToRedis?
I’m assuming that otherwise Meteor would still have to process the whole oplog if any one publication needed it.
so if I wanted to add support for batch operations I could wrap my rawCollection access with something that would also issue the appropriate INSERT or UPDATE events.
Yes, you could wrap it, or could create a service that calls it and pushes to redis.
And to get the most performance out of this would I basically have to convert all of the existing publish calls with publishToRedis?
Yes. But you won’t get “wow” performance, just by doing that. Trully high performance is gained when you fine-tune the reactivity, namespaces, channels, “direct ids”. I did not run benchmarks yet, I’m curious also to see. But I can only compare oplog with the “non-fine-tuned” aspect, otherwise it won’t be fair.
I’m assuming that otherwise Meteor would still have to process the whole oplog if any one publication needed it.
True
Does the client side subscription code have to change at all to use the redis oplog?
@techplex client-side, absolutely no change.
You can even do RedisOplog.init({ overridePublishFunction: true }), and server-side code doesn’t have to change either. Same old Meteor.publish returning cursor/array of cursors.
2 Likes
is doing a RedisOplog.init({ overridePublishFunction: true }) on an existing automatically change to Redis Oplog without doing any changes to the codebase?
I think I know the answer to this, but just to be sure, I’m going to ask the question…
I assume we can’t currently use this for any publications that are built using the reywood:publish-composite package?
I’m hoping it’s possible (or will be possible in future) as I really want to try this out, but my app relies on publish-composite quite heavily!
Great work though @diaconutheodor 
3 Likes
Thanks!
Yes you are correct, no publishComposite yet. And I am really aware of this need. At first I really wanted to do it, I even began drawing the schematic for it. But then I realized it’s super hard to make a reactive-graph performant, I needed to put it in the background of the mind to let it mature. + I really wanted it for “Grapher”, which uses publishComposite as backbone for the reactive query graph.
I studied what publishComposite uses, the reason that makes it hard to just port it, is because he uses cursor.observeChanges + It is so very unperformant, whenever they detect a change to a doc, it republishes all the children just by looking at the code, I cannot simply port such a thing into a library that is focused on performance and scalability.
I will have to rewrite it. At this point I think I know what to do, won’t promise dates, but it is solvable and can become super performant.
8 Likes
Ok guys, so in order for this to get the desired adoption, we need publishComposite, for sure, it’s not optional anymore and I understand this.
Because most of the apps that are in production use publishComposite (including ours), we cannot simply say “just rely on oplog for publish composite” or polling!
The problem of publish composite is, ofcourse, performance. It’ll be a CPU & Memory hog:
For: users -> posts -> comments. We will have to instantiate 1 observable collection for users, lengthOf(users) observable collections for posts, and lengthOf(allPosts) for comments. If you have 10 users, each user has 10 posts and each post has 10 comments. We’ll have: 1 + 10 + 10*10 = 111 “publications”. Which is going to be roughly the same of having 111 subscriptions. Crazy.
The first idea I wanted to implement was doing an aggregation of filters at each child-node. Example, instead of creating lengthOf(users) independent ObservableCollections for posts, just do a single one, where filters represent all filters in an $or array. This idea had it’s birth here: https://github.com/cult-of-coders/grapher/issues/36 . Do what we do for Grapher non-reactive fetch (assembly and deassembly) do for publishComposite.
The problem is that it can work with Grapher, because the way of getting links is known, the fetching strategy is immutable so we know how to choose whether we use aggregate framework or not. However, in this case, publishComposite would give users ultimate flexibility in terms of displaying any type or related data.
Aggregation of filters concept will fail if we have “limit”, “skip” in child cursors. We cannot fetch’em all in a single query. Especially when these can change in certain children, because we have a function for each child element.
The options I see now:
- No limit/skip in child nodes (So we can aggregate the filters)
- Implement it bare-bones like I described above. (It can still be more performant than the current publishComposite)
- Move the composition client-side. (The problem with this is that it can lead to a lot of security wholes and a lot of boilerplate code, I know this because I solved this issue in Grapher using body exposure, it’s crazy how deep things can get)
- What if we could stop reactivity at a certain node ? Example I want all users with all posts with all comments. I maybe don’t want reactivity for comments. Or I want it only for users, or only for posts and comments. The higher the reactivity stops, the faster it is. But then the question arises, why use publish at all if you want to stop reactivity ? Doesn’t seem like a good thing.
I will continue with option #2 now, just to have the ability to say farewell to MongoDB oplog.
2 Likes
That sounds like a great place to start - any improvement in performance is good progress!
Really appreciate the time and effort you are putting into doing this. I hope to be able to contribute toward the project very soon.
1 Like
Publish Composite is theoretically done. I stopped re-inventing the wheel, I can’t make it as performant as I wanted, however, we can use “channels” and “custom namespacing” at each node level in the publishComposite. Managed to hack my way and use the same code-base as publishComposite and we have the ability of fine-tuning our reactive graph 
Right now I just need to fix the tests, something weird is happening when using console-tester, it seems that methods are called twice. Very weird
3 Likes
I tried this plugin with a Stock Exchange meteor app and it works incredibly!
When I have some spare time I will make a load test and read the code, there are a few things I don’t understand with this module.
Redis on a single core SSD server can do 40.000 ops/sec,
on the same server MongoDB with WiredTiger can do 15.000 insert/sec and 25.000 query/sec.
This is not orders of magnitude faster, why don’t use MongoDB without oplog,
or write a Memory module for Meteor and store data inside the Meteor server?
What is the reason for choosing Redis, because is it more lightweight, or because of the pub/sub?
It seems to me this module will make apps scalable not because Redis a little faster than MongoDB, but because switching off Reactivity.
It is fine if you use only 1 Meteor server, but if you have 2 or more the users will see different state depending on which server they are connected to.
If a user on server1 deleting a post, Redis publishes to all users connected to server1 that this post has been removed, while the server2 won’t get notified about this deletion by the central MongoDB because there is no more oplog, and the server2’s Redis still publish the deleted post to its users.
I made a manual insert into the central MongoDB and the Meteor server got this new document, now I don’t understand it even more, if the Meteor server get all new documents from the MongoDB how this will solve the CPU issues?
Hi, thanks for trying it out, really appreciate it.
Redis on a single core SSD server can do 40.000 ops/sec,
on the same server MongoDB with WiredTiger can do 15.000 insert/sec and 25.000 query/sec.
Redis pub/sub system can handle even more. I don’t know exact numbers, but around ~300,000 msgs/s ? Chose Redis bc it’s mature, stable, scalable and fast. Any pub/sub system would have been fit. Redis was the popular choice.
It seems to me this module will make apps scalable not because Redis a little faster than MongoDB, but because switching off Reactivity.
Not only, it’s because we now have the ability to fine-tune reactivity, I have given the example of a chat thread, in which your “listener” only receives the messages from the chat thread (so it does not process everything) and ofcourse ability to create “SyntheticMutations” a new concept that is briefly described in the README.md.
If a user on server1 deleting a post, Redis publishes to all users connected to server1 that this post has been removed, while the server2 won’t get notified about this deletion by the central MongoDB because there is no more oplog, and the server2’s Redis still publish the deleted post to its users.
Ok so server1 and server2 communicate with the same redis server. A change in server1 is published to redis. And if server2 has an active publication, thus subscribing to a redis channel, it will receive the message, and process it.
I made a manual insert into the central MongoDB and the Meteor server got this new document, now I don’t understand it even more
Something is very wrong here care to show me a reproduction repo ? Basically if you change the MongoDB outside the app, those changes will not be reflected unless it’s somehow polling, or you made an update that triggered a requery. (Sometimes we have to requery the docs of a publication especially when we’re dealing with limit/sort cursors)
Thanks!
Does this work apply to apollo? So far apollo did not implement mongodb tailing.
Do you have any plans for an apollo implementation?
1 Like
It can be integrated with apollo yes. Not the main focus right now.
So using redis oplog hosted in digitalocean, MongoDb on Compose.io and several Galaxy containers as the app server with no out of meteor database change will reach to higher performance.
Galaxy does not have redis so there is a little network latency but I think it has a little impact.
I think/hope it is a defacto standard and best practice.
I don’t use redis in compose because of its pricing, digitalocean is cheaper and redis does not have MongoDb devops.
Does anybody have better solution?