Meteor Scaling - Redis Oplog [Status: Prod ready]

hats off to you both, double hat to @diaconutheodor

3 Likes

Is your app reactive? Are you using polling maybe?

not really since all my queries are all based on my subscription params.

Redis-oplog is a package to replace oplog trailing for reactivity. If you want your app to be reactive in a scalable way, I highly recommend it. I havenā€™t used polling before, so not sure if the non-oplog query is polling by default.

1 Like

Thank you @ramez that makes sense to put meteor and mongo together because of the memory and cpu differences. It would be awesome if you could add a sanitized version of your redis and mongo scripts to the github pm2 repository you created.

Do you mind taking a look? The pm2.json there already includes both. Anything else I could add to make it easier?

Yes I was looking at the pm2.json, I thought there might be startup conf file for redis or MongoDB that you use, I have so little time to learn the devops stuff I am mostly programming, so I appreciate any information. You have already helped me a lot with all the information you have already posted.

Got it, let find some relevant stuff from our knowledge base and add it to the repo readme (or maybe another md file)

Well I tried to go to Prod yesterday but ended up with some edge cases where on some updates it would make the change and save it in mongo and send the change to redis all like expected, but then the client would revert back to the old version. I am doing some more debugging, but if anyone might have an ideaā€¦

@jamesgibson14 what kind of updates, what kind of publications did you have ? Try to reproduce it locally and submit an issue to GitHub. Thank you!

Well it seems to be problems with cursor.observeChanges and also cursors that have `fields: {name:1}'
I am working right now on local to see if I can isolate the problem.

This was one of the modifiers: { '$addToSet': { passengers: { _id: 'y2MECXDgr9ggiP5D4', name: 'Marlee Nielsen', phone: '' } } }

1 Like

I think it is most likely problems in my code.
Another thing I noticed was as I have Redis in debug mode I was getting a lot of Redis Events that I donā€™t think I was subscribed to. An instance shouldnā€™t get any redis events if they are not subscribed to a channel right?

@diaconutheodor verified redis worked by using redis-cli, thanks a lotā€¦

@ramez How to use to authicate users with casperjs, I have tried casper.setHttpAuth, however it doesnā€™t work.

@diaconutheodor is there a way to see all subscriptions to channels in redis-oplog from meteor shell?

We inject js and call Meteor.loginWithPassword - want me to share our code?

that will be greatā€¦ TIA!

@ramez are you using meteor-casperjs (https://atmospherejs.com/nblazer/casperjs)? if so, then test clients have to be meteor, which are quite bulky for stress testā€¦

No, regular NPM casperjs. So pure nodeJS code. Sorry for delay, Iā€™ll upload tomorrow on Github. I just need to sanitize and document.

Thatā€™s ok, take your time. Thanks a lot

Here is the link:

Pls PR / raise issues if things fail. Thanks and good luck

2 Likes