Threads like this make me love the Meteor community even more!
So much love guys! 
And of course a special thanks to @diaconutheodor for all the hard work he puts into RedisOplog and Grapher!
(Sorry for OT)
Is there any reason why redis-oplog would cause subscriptions to stop and restart when an field on the logged in user changes. I’m currently seeing a strange issue when using my socialize:user-presence package.
The package works by allowing registration of callback functions that get called when the user connects/disconnects or goes idle. I’ve registered the callbacks to set the status filed on the user record. When that field changes though it causes the subscription on the client to stop and then start again, which changes the status field again causing an infinite publish/subscribe loop. If I remove redis-oplog the issue goes away and every thing operates as intended.
@copleykj there’s an issue that’s bugging me and couldn’t reproduce:
I think this is it, they use user-presence. Once I manage to reproduce it I can fix it in minutes. Can you help me with a reproduction repo on that ?
I am now trying to see if I can do it with user-presence, also the test I wrote is here:
https://github.com/cult-of-coders/redis-oplog/blob/feature/redis-vent/testing/accounts/client.js
Update:
I reproduced in the test. Nice!!! Finally. Solving it soon.
Sure, https://github.com/copleykj/meteor-socialize-demo is the project I’m experiencing the issue on. The project uses packages that aren’t on atmosphere as of yet so there is a script to facilitate cloning the necessary packages. Details in the README.
@copleykj I found it, thank you so much, I notice now the infinite loop executing onUserOnline over and over again if I do:
UserPresence.onUserOnline(function(userId){
Meteor.users.update({_id:userId}, {$set:{status:"online"}})
});
I will dig into your packages and see why 
Yes, specifically the publication in socialize:user-presence is being stopped and started repeatedly.
Meteor.publish(null, function userPresenceSessionConnected() {
if (this.userId && this.connection && this.connection.id) {
userConnected(this.connection.id, this.userId, ServerPresence.serverId(), this.connection);
sessionConnected(this.connection);
this.onStop(() => {
userDisconnected(this.connection.id, this.userId, this.connection);
sessionDisconnected(this.connection);
});
}
this.ready();
}, { is_auto: true });
Updated to 1.2.4
- We have cleared all the (known) bugs
- We worked on improving performance even more.
- We added the Vent mechanism. (https://github.com/cult-of-coders/redis-oplog/blob/master/docs/vent.md)
7 Likes
Thanks so much for this update! This fixed my issue with the presence package.
I’m seeing one additional issue though with optimistic updates on inserts. Upon client side insert I see the UI update to display the new document, then the UI updates again with the new doc removed and then finally back to being displayed again. You can see this behavior in the demo by creating an account, signing in, and creating a new post.
@copleykj optimistic ui is done in the method ? If yes take a look here: https://github.com/cult-of-coders/redis-oplog/blob/master/docs/optimistic_ui.md
Its a client side insert operation.
Ok, please create these issues on GitHub, so we have a centralized place for them I’ll take care of it.
1 Like
Sure thing, headed there now 
Reasons to be excited:
Simon from classcraft:
First, thank you very much for giving this a chance, the biggest Meteor app is the best first case study. I am going to be on top of it to make sure it’s running smooth for them.
Must admit, got a bit excited, an almost 100% improvement is not to be ignored. But honestly, I would’ve been much happier with a 300% improvement. Anyway, 100% doesn’t necessarily mean that you can run on half of your infrastructure, because the more you grow the better the performance will get. And the important part is: you can continue growing
The fight isn’t over yet
What if I told there is room for even more performance improvement ? (https://github.com/cult-of-coders/redis-oplog/issues/199) But this one is relatively hard… It’ll take me a while to accomplish and do it right (At least 16 hours of work), as I need to rethink the architecture and refactor stuff so it doesn’t become an unmaintainable mess.
Just a little bit of a history since we’re celebrating a nice achievement:
- First we did the fetching (at the insert level) when we mutated and push it to redis
- Then we realized it’s prone to race-conditions so we moved fetching to processors that interogate the db individually
- Then we realized if we have a lot of observers, it can lead to 30-40 db requests for a single event
- Then we optimized and aggregated what is needed for processing into 1 single db request / event (unless it’s a removal event)
- Then this is the holly grail: issue #199
I won’t even start on how many hairs I pulled because of Optimistic UI !! But we nailed it in the end.
Stressing some things out:
The road to making any DB reactive has been opened.
The road to scaling reactivity infinitely has been opened.
The road of heavy investments into big Meteor apps has been opened.
We made this happen together, it was a shared effort.
23 Likes
That’s some crazy news
I have an interview blog post with Shawn Young, the CEO of ClassCraft on the way - hopefully it will be ready next week. They are running one of the largest Meteor apps - and I can tell you, they are not going light on pub/sub - so this is really big news.
The fact that they cut down their container use by half is a great reason to use RedisOplog even if you do not “need” it. I’m feeling bad for MDG revenue numbers though. Either way, I think this package a net gain for them, and I hope they start to recognize it.
2 Likes
Short term loss for long term gain, success stories at scale will attract more enterprises for sure
1 Like
I’m not feeling bad at all… on the contrary, many people move away from Galaxy because of the cost, but now if you can handle many users with just 3 containers then it’s already a plus, no more devops, integrated APM, and hopefully soon, integrated Redis!
3 Likes
I don’t know what MDG’s plans are, but frankly this work needs to be compensated and integrated into Meteor with official backing. And collaborate on combining this with any upcoming changes to Meteor for Mongo 3.6 or Apollo integration - the fact that we still don’t have clear answers on if or how MDG plans to combine Meteor and Apollo is a bit baffling.
And someone needs to offer up a ‘Meteor stack’ on AWS marketplace - with built in redis-oplog, Kadira, and scripts for deployment and autoscaling. It will be a more powerful solution than Galaxy not to mention cheaper.
3 Likes
@diaconutheodor I want to thank you for this from the bottom of my heart… This level of scalability has been on my wishlist since the 1.0 release. I have a lot of new found enthusiasm for Meteor and it’s future now and that speaks volumes.
6 Likes
Hey guys,
I’m the developer working on the redis-oplog integration at Classcraft. As @diaconutheodor mentioned we released the package integration last night and we’ve seen a significant improvement of CPU usage.
After a full day of collecting data it looks the improvement might be less than the 100% I mentioned to @diaconutheodor this morning but it’s still significant. And I’m still working on namespaces/channels optimization so it’ll only get better. I’ll keep you informed.
We’re super enthusiast about this and I’m sure we’ll be able to contribute to the project. The future of Meteor is definitely in the hands of the community.
Kudos to @diaconutheodor and everybody who made this possible.
15 Likes
At first I was feeling a bit bad about such low performance improvements (it was bittersweet), but then I realized that at that scale, you will head into the same problems oplog had without namespaces/channels/vent: Too much stuff to process. And RedisOplog requires more CPU to process stuff than classic Oplog, because the data that comes in needs to be fetched from DB to avoid race-conditions.
But this is where RedisOplog truly shines: in the fine-tuning, including the newest addition: redis-oplog/vent.md at master · cult-of-coders/redis-oplog · GitHub . So all in all, just by adding it and gaining almost 100%, even 50% is not that bad.
@copleykj glad you have your enthusiasm back. I am very happy with how Meteor is evolving honestly. I still see nothing better. But I see some stuff we can steal from NextJS.
We already backed (Feature Request: Deploying a Meteor App with AWS Elastic Beanstalk · Issue #768 · zodern/meteor-up · GitHub) to support elastic beanstalk aws deployments, setting up kadira and redis yourself doesn’t seem like that big of a deal honestly.
Apollo and Meteor seem to me 2 different beasts. Apollo is npm, you can already start using it in Meteor, what level of integration are you looking for ? (Just curious) This seems more than enough to me: https://www.apollographql.com/docs/react/recipes/meteor.html
If you’re looking for GraphQL like queries, take a look at Grapher it is my vision for the evolution of data fetching layer, inside Meteor, and I hope that it will be recognized one day as the default way to fetch data from MongoDB inside Meteor. I see it as a bigger achievement than RedisOplog (but some of you may contradict me) @simonbelanger you should take a look at Grapher also, it may be the next stepping stone for Classcraft.
2 Likes