Meteor Scaling - Redis Oplog [Status: Prod ready]

ignl · November 16, 2016, 1:53pm

Great work, I will definitely will try it on my app. Thank you for taking time to do this. Meteor really needed something like that!

aadams · November 16, 2016, 2:32pm

I’ll try it out too when I hear the word its production ready.

aadams · November 16, 2016, 2:35pm

Ok, maybe not on our own just yet. I think Benjamn is only working the build tool, updating mongo drivers, bumping node versions, and bug fixes outside the core tech. Core tech we really care about, like scalability capabilities, Livedata, DDP, Tracker, Sub/Pub, Blaze is now outside MDG’s scope and are community responsibility.

That’s why it’s so surprising and exciting that someone has picked this critical issue up, some thought Meteor as it stood 2 months ago was fully baked in terms of core tech and scaling capabilities.

shawnyoung · November 16, 2016, 2:53pm

Just found this thread – boy, am I excited! I’m the founder of Classcraft, we have just shy of 2M users. Our entire app is built in Meteor, hosted on Galaxy. As you can imagine, scaling has been a challenge

Thanks for all the hard work @diaconutheodor, this looks really promising. We’ll look into it over here and get back to you.

diaconutheodor · November 16, 2016, 3:09pm

@shawnyoung thank you. please, if you want to help, plug it in your development, and try to find some bugs, it’s just plug-in (5min work?). And see if something breaks. It will be of an imense help for this, really. (PS: 2M is very shy indeed) You can help with some ideas since you have a very deep insight of scaling and making meteor performant, check this out: https://github.com/cult-of-coders/redis-oplog/issues/21 )

@aadams

That’s why it’s so surprising and exciting that someone has picked this critical issue up, some thought Meteor as it stood 2 months ago was fully baked.

They did what they have to do. They made this gorgeous thing open-source, for me it’s enough to raise my hat off to this people and thank them and help them however I can. Meteor is not such a big framework (ok it’s huge, but let’s not compare it to ruby on rails, symfony2, zend framework) 3-5 hard-core guys can maintain this.

I’ll try it out too when I hear the word its production ready.

It takes you just 5 mins to plug it in, without modifying your code at all, and play with your app to try to see if it fails. It would be EXTREMELY helpful, more people that test this, more scenarios it can break. Yesterday someone had an issue with positional operator update. I fixed it in 30min, it was super good.

aadams · November 16, 2016, 3:15pm

Ok, I’m using mlabs as my MongoDB endpoint, will I need to do anything on that side?

Where do we init inside the application (as per your Usage heading)?

I need to change out my publish methods for publishWithRedis?

diaconutheodor · November 16, 2016, 3:34pm

@aadams

Do you use mlab as local db ? Shouldn’t matter though. Don’t put it in production before you test it! If you want to test with your remote data, copy it in a local db. Thanks

You don’t need to override your publish methods. Do like this:

meteor add cultofcoders:redis-oplog

Install redis locally.
Before you load anything server-side. You need to instantiate it.

Create a redis.js file and import it before anything else is loaded.

import { RedisOplog } from 'meteor/cultofcoders:redis-oplog';

RedisOplog.init({overridePublishFunction: true})

aadams · November 16, 2016, 3:44pm

I have a development mlabs db that I connect to from my local box – yes.

Huh? If you care for me to test it, on my dev and staging boxes, you need to be specific about what I should do.

Do I have to install Redis locally if I use mlabs for dev? And how would this look if I wanted to push it to my staging EC2 server?

I understand, this makes sense.

diaconutheodor · November 16, 2016, 3:54pm

You can install redis anywhere. You have the ability to configure your redis server. Check the README.md file.

aadams · November 16, 2016, 4:42pm

Yeah, this is where I’m getting confused, it’s not clear from the readme what to do about the redis server (either from my dev box or from a staging server). Things have to be so simple to gain adoption. I’m someone that knows NOTHING about redis.

diaconutheodor · November 16, 2016, 5:00pm

Sorry about this. I will try to make it more clear on the next release. There was a lot of confusion on how this works. I need to write some diagrams, better docs, and invest time in making something that is well documented.

aadams · November 16, 2016, 5:15pm

Sounds good, I’ll wait until the next release to check it out again.

arggh · November 16, 2016, 6:09pm

Sorry about this. I will try to make it more clear on the next release

I found the instructions pretty straightforward (as long as one understands how Oplog works and knows what Redis is). Actually this is so straightforward, I’m kind of surprised nobody came up with this idea until now!

That being said, a diagram might be a good solution for “drawing the bigger picture” and visualizing exactly where in the stack this package works it’s magic.

aadams · November 16, 2016, 6:22pm

I also don’t know how Oplog works in any great detail.

Your post seems to be a dismissal of my post. I hope this doesn’t mean this tool is not for guys like me.

arggh · November 16, 2016, 6:50pm

Your post seems to be a dismissal of my post

This was not my intention, so I apologize. I was trying to imply, that simply providing a diagram of where the “trick” of this package takes place in the stack might do the trick.

diaconutheodor · November 16, 2016, 7:31pm

@arggh

Actually this is so straightforward, I’m kind of surprised nobody came up with this idea until now!

See how this whole thread started, I was thinking the same at first, that there may be some other underlying issues, but there aren’t really. It just works

@aadams

I hope this doesn’t mean this tool is not for guys like me.

arggh just gave his feedback that’s all, and it’s good to know some people understand some people don’t. Seems that someone who understands how oplog works understand why this is helpful and how to use-it, someone who doesn’t may need some additional explanations. Good feedback!

This would be a tool for any level, by the time I’m labeling production ready I will make sure anyone understand what it does, how it does it, and how we can use the new concepts of namespacing, channels and synthetic mutations. Until then try to break this!

I believe the best way to illustrate this would be to write a small chat app using channels and synthetic mutations, put it on a 10$ digital ocean instance, and stress test 500 people talking to each other.

Longmate · November 16, 2016, 8:51pm

@diaconutheodor

You were right! I had a couple of other imports running before Redis.init(). So I moved the Redis Oplog code into a separate module and imported it at the very top of my server entry point and I can now see everything working including the debugging messages in the server console and Redis monitor

Done a few tests within my app across a couple of instances and it seems to be working perfectly! I’ll persist trying to break it though

I have found one small issue relating to the ‘overridePublishFunction’ option… When I had this set to ‘true’ it appeared to try to import my first publication repeatedly and then a call stack size exceeded exception was thrown (see console output below). I managed to get around the issue by switching ‘overridePublishFunction’ back to false and renaming all the Meteor.publish references in my app to Meteor.publishWithRedis.

/usr/local/bin/meteor --settings settings-dev.json
[[[[[ ~/Omono/omono ]]]]]

=> Started proxy.
=> Started MongoDB.
I20161116-20:36:06.927(0)? Starting Redis OPLOG :)
I20161116-20:36:06.982(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.982(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.982(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.983(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.983(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.983(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.983(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.983(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.983(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.984(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.984(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.985(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.985(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.986(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.986(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.986(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.987(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.988(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.988(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.988(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.988(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.989(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.989(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.989(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.989(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.989(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.989(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.989(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.989(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.990(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.990(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.990(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.990(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.990(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.991(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.991(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.992(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.992(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.992(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.993(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.993(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.993(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.994(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.994(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.994(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.994(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.994(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.995(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.995(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.995(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.995(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.995(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.995(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.995(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.996(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.996(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.996(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.997(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.997(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.997(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.997(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.998(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.998(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.998(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.998(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.998(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.998(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.998(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.999(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.999(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.999(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.999(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.999(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.999(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.999(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:06.999(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.000(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.000(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.001(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.001(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.001(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.001(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.001(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.002(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.002(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.002(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.002(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.002(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.002(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.002(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.002(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.003(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.003(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.003(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.003(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.004(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.004(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.004(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.004(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.004(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.005(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.005(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.005(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.007(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.008(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.008(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.008(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.008(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.008(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.009(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.009(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.009(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.009(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.009(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.010(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.010(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.010(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.010(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.010(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.012(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.013(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.013(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.014(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.015(0)? [Main] Created publication with name: Addresses.Company.List
I20161116-20:36:07.015(0)? [Main] Created publication with name: Addresses.Company.List
W20161116-20:36:07.022(0)? (STDERR) /Users/chris/.meteor/packages/meteor-tool/.1.4.2_1.qrj6e3++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:280
W20161116-20:36:07.031(0)? (STDERR) 						throw(ex);
W20161116-20:36:07.032(0)? (STDERR) 						^
W20161116-20:36:07.032(0)? (STDERR) 
W20161116-20:36:07.032(0)? (STDERR) RangeError: Maximum call stack size exceeded
W20161116-20:36:07.032(0)? (STDERR)     at module.export.exports.default (packages/cultofcoders:redis-oplog/lib/debug.js:5:9)
W20161116-20:36:07.032(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:31:5)
W20161116-20:36:07.033(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.033(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.033(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.033(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.033(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.034(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.034(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.034(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.034(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.035(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.035(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.035(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.035(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
W20161116-20:36:07.035(0)? (STDERR)     at Object.publishWithRedis (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:33:12)
=> Exited with code: 1

diaconutheodor · November 16, 2016, 9:05pm

@Longmate

Nice! This is very good feedback! Glad to hear it works good for you.

Hmmm the infinite loop may be due to the fact that somehow redis is initialized twice thus, extending publish method twice, thus leading to an infinite loop ? Can you check this ?

If you have publish composite it should work out-of-the-box. Your post gave me hope, maybe we can have it stable before end of year. So we can start 2017 with a BANG.

Spyridon · November 16, 2016, 9:06pm

I have not tried this yet, the only app I could try it on is our live business app which I can’t take risks on atm, but I have a question regarding this.

You say that you can install redis anywhere, does this hold true for apps deployed on Galaxy though? Deployment there is automatic, so it’s not like we can really choose what is installed. Is there any way to get this working on Galaxy?

If not, that’s something to think about long-term!

diaconutheodor · November 16, 2016, 9:09pm

@Spyridon you could try it first locally, right ?

Yes. You can set it up in an AWS in an EC2 instance in the same region as your Galaxy containers. And simply configure the ip. It would be ideal to have it somehow in the same network, not sure if Galaxy supports this.

RedisOplog.init({redis: options})

Options can be found here: https://www.npmjs.com/package/redis#options-object-properties