Granted, Hansoft developers contribute back to the Meteor Core solving issues they spotted: https://github.com/meteor/meteor/pull/4694
1 Like
It looks like the commit bears directly on the discussion in the video. However the confusion I have is in the talk being from August 27th yet the link indicates their fix was merged on July 7th.
Perhaps someone in the know can summarize outstanding performance issues with Meteor affecting vertical scalability. From reading the Livequery documentation, it somewhat hints that it’s a ‘work in progress’ and there is still work to be done.
Indeed. My worry is the ‘work to be done’ will be proprietary 
1 Like
A little birdie told me Hansoft’s ultimate solution did involve a custom Redis backend. I’m more interested in meteor and vertical scaling sans any jury rigging.
Interesting. It seems like once RethinkDB is supported we won’t have to resort to hacks like oplog tailing and Redis to scale Meteor? I hoping that will alleviate some pressure on Meteor.
As a community it also might be worth looking into how Phoenix multiplexes channels to see if we can gain some insight for Meteor. Granted Phoenix is running on the Erlang VM and Node will never scale to millions of connections per server but perhaps we can grok something?
Well, somewhere in the stack some component will have to process updates to the database anyway. Putting it in the DB itself will probably be more efficient than Meteor’s oplog tailing, but there’s no such thing as infinite scaling.
Not millions for sure, but thousands per node should not be out of the question. However http://vertx.io/ more than Pheonix likely has ideas you can apply. It’s more conceptually congruent, being a polygot node implementation on the jvm.
There is no infinite scaling true, but I would be excited to see Meteor scale up to the point where SkinnyGeek1010 laments with an angry fist the day he decided to like the idea of immutable state stores.
2 Likes
I agree that letting the DB handle this would be much better, like RethinkDB for example.
1 Like
Any update on this?. We’re worried our production app could reach limits soon and moving to galaxy is not an option (no eu aws support yet).
How are big meteor apps managing this limit?
Separate databases (ie. only hold ready, consumable data in the MongoDB that your Meteor app connects to). The other thing suggested by @debergalis is to only observe certain collections and put some kind of oplog filter in between your Meteor app and database. That could make for a good project.
This is a concern I share, however assuming all writes succeed is not the answer in my opinion. Personally, RethinkDB seems like the best solution on the horizon.
One possible solution is to swap out the subscriptions for a meteor method if it doesn’t have to be instant real time. Long polling can be cheaper in some cases. This worked out fairly well for me but it depends on the project of course. Typically the Meteor method would do the same sort/query that the template helper would do in Minimongo. In this case if there was new data you would just use the entire response as is.
Postgres, RethinkDB and Redis all have some sort of pub/sub built in.
Stack Overflow uses a mere 9 web servers to support 260k+ concurrent users with tons of reactive data, all being fed from Redis’ pub/sub. Their biggest issue is running out of sockets at the os level.
It makes the silly Blaze thread quite mind boggling. What exactly is the point of fretting over which front-end to stick onto a back-end that doesn’t really work once it is actually asked to work?
@zimme says that Hansoft really want to give their solution back to the community. Hopefully we’ll see it one day. Much as @msavin suggested, it involved inserting an intermediary between Meteor and the database.
For sure many things wouldn’t be noticed if it were not sub millisecond. It seems though with meteor everything by default is assumed to be critically reactive. I see people in irc and the forums using mongodb and pub/sub for chat messages.
Perhaps that’s a function of the docs and the way they present things.
3 Likes
I’ve been looking into how Phoenix handles it’s realtime ‘channels’ and they approach it differently. This isn’t going to get Meteor 2mil websocket connections but it seems to solve the livequery pressure.
It actually works without any database if that was needed. Instead of trying to sync to the database and tail an oplog the approach for a public chat room would go something like this:
the last three are the most interesting
- server listens for a
"lobby:join"event - client loads JS library and connects with one socket and multiplexes different ‘channels’
- client calls
"lobby:join" - server handles
joinevent (and opt. authenticates) - server
after:joinevent fires and does query for chats and returns to client to prime their data - client fired
"on join"callback and uses resp. data (for Blaze store in Session for temp. helpers?) -
any client subscribed to the channel would send
"msg:new"to server - server would handle
"msg:new"by taking new message and pushing it down to each subscriber tolobby - each client
on msg:newwould push the data onto their prev array of data.
The data gets pushed down as new data is sent up to the server. You’re only listening to users connected to that ‘channel’.
Also, if you’re using Rethink or Postgres you can have the server listen for triggers on a query and then have the server itself push down a "msg:new" instead of relying on the client to send up a changed event.
However one issue, using a DB without query triggers means you can’t directly do an insert/update on the client and have reactivity. IMHO doing a "msg:new" instead of Message.insert() isn’t a big loss for me and it also makes it trivial to use other databases.
1 Like
Why not just simply use Redis to cache and publish all the events while they are written asynchronously on MongoDB ?
Example :
- Client calls method : ‘msg.insert’ : the method will insert the msg in MongoDB and in Redis.
- The publication on the server will subscribe to Redis for a particular room and send an ‘added’ event.
This method can easily scale and uses Redis as it should be used.
(Of course, we’ll need to imitate the DDP protocol with Redis to add different events like : ‘changed’ and ‘removed’)
2 Likes
So a standard observer pattern. However one thing you’ll need to consider when horizontally scaling is to have a single node or cluster of nodes handle all traffic for a particular channel. That complicates your front end proxy, but only slightly.
Otherwise you end up with something more resembling an eventbus where all data is just dumped onto a shared bus. That works ok in a stateless rest environment where the first node with time to act on it removes it from the bus. Does not scale when you’re trying to synch state across nodes.
Check out the above video at 21:00, where they describe using Socket.io and Redis in a similar manner. When they got to concurrent user 1001 the entire setup collapsed. Adding more nodes just made things worse. Saturation city!
3 Likes
That’s really interesting! And frightening! It seems like Node is really hard to scale in general 
Luckily I haven’t needed web scale yet 
Yea I was wondering how that would work across nodes. One thing I found out is they don’t require sticky sessions. I dug into this further and they’re using the Erlang VM’s ETS in memory database to handle this somehow. If you’re using Phoenix on Heroku there’s a Redis pub/sub connector. I wish I could dive further into how they’re doing this but i’m getting in over my head now 
I also asked the Phoenix author about how you would handle getting out of sync if you’re only listening to those client messages:
[Me] How does Phoenix handle clients getting out of sync using channels? It seems like if you add a database to the chat example it’s possible for the client to get out of sync?
[Chris]: you’d keep a “last_seen_id” on the client
on first join to a channel (app boot), you pass down a list of messages to sync the client up
subsequent joins (dropped connection/channel crash/reconnection), you pass up the last_message_id you saw
then your join query only sends messages since that
2 Likes
I talked about something like that a while ago but nobody answered or gave his opinion :
1 Like
I imagine something like Redis or RabbitMQ would be needed. I’ve heard good things about MGTT being used for this too.
I’m actually experimenting with something like this now… should have the experiment open source soon!
I’m wondering what the limits are for just opening up a publication (as in how many can one server handle)? And are multiple publications on a single socket? (i’d assume so).
2 Likes
As long as your requests are totally stateless (REST) you can scale node like crazy. Any particular node doesn’t know that any other nodes even exist. The problem with oplog tailing is that every node is de-facto observing the total state. You end up saturating cpu really quick when there is a write.
Unfortunately in the video he doesn’t mention the limit they reached when switching from socket.io to socks.js and Redis pub/sub. The final solution in the video was a high performance db proxy written in c. I’m not sure what Hansoft’s db proxy is written in, but he did harp about how most of them are “C++ Gods”
Zimmy was however indicating that the Hansoft changes were working fine at up to 10k concurrent connections per meteor node. While that’s nowhere close to running out of ports or file descriptors I’d be still happy with that!
Here’s a talk on how StackOverflow uses Redis (mostly live coding!). He compares SQL with Redis at one point with SQL handling 9k ops while Redis churned out 19k ops.
Redis is fast. All in memory and literally hand tuned to take into account things like chip and instruction cache and whatnot. The eventual limits of Postgres or RethinkDB’s pub/sub I have no information on, but it won’t be close to Redis.
The guy in the video sure answered your question. The answer was… err no! At least not with socket.io.
3 Likes