Hi folks - I thought I would do some market research for MDG Currently, Meteor allows us to write reactive database queries by tailing MongoDB’s operation log. However, MongoDB recently announced a feature called Change Streams which paves the way for a more scalable but limited solution.
LiveQuery (Current Solution - aka oplog tailing)
Meteor observes the operations log of the entire MongoDB database and looks for changes. When changes happen in the database, Meteor notifies clients of them.
Pros: database queries can be reactive
Cons: expensive, limited scale
LiveDocument (Proposed Solution)
MongoDB can notify the Meteor server of changes using Change Streams. This is more efficient than sifting through logs but has limitations. MongoDB Change Streams can only provide updates on specific documents. It cannot let the application know that the documents in a query have changed.
However, maybe we can use it with publications that use findOne, and in general, recommend people to fetch find queries via Methods instead.
Cons: not suitable for database queries
Pros: better perform and greater scale
Solution
Meteor should implement reactivity for individual documents subscriptions and not queries.
Reasoning
Based on experience and what I’ve read on the forums, I can say:
Methods are much faster and more efficient at fetching large queries
Pub/sub is much faster and more efficient when subscribing to one document
Thus, maybe developers who are focused on scale can disable LiveQuery, use pub/sub for single document queries powered by “LiveDocument”, and use Methods/Apollo/etc for larger data sets.
What do you think?
I am fine using Pub/Sub as it is for reactive queries
I am for using Change Streams + Pub/Sub for one document queries and using Methods/Apollo/etc for the rest
I will use a different real-time database (like Redis or RethinkDB) and/or data layer (like Apollo) in the future
This could be a good play for Meteor + Apollo, as Meteor can become a fast and efficient framework for real-time data, while Apollo can be used for more complex queries.
This would also pave the way for the much-needed revamp of the Accounts package. If it’s not reasonable to get rid of the account’s subscription, at least it could become more efficient.
Maybe this combination would ease the “ditch pub/sub in favor of Apollo” camp.
@msavin right now I’ve a newsfeed with posts and comments (very typical social networking feature). The newsfeed subscribes to the posts publication and when the user clicks on view comments, the client will subscribe to the comments publication specific to that post Id to share the users comments in realtime.
I believe this is a very typical use case for realtime (in addition to chat). Speaking high level, how can we re-implement this use case using your second option (Change Streams + Pub/Sub for single doc) in your opinion? Any pros/cons?
Good question. There’s a lot of factors in play, but here are some ideas to get the wheels turning:
In the case of a newsfeed, assuming you are using the “mailbox” model, you can create a LiveDocument that tells you when the mailbox had been last updated. If the last updated time is greater than the last time your client fetched data, then your client can re-fetch. (Tracker + client-only MongoDB collection can be used for this.)
In the case of comments, you could keep all the comments and responses inside of one MongoDB document.
In the case of a chat, you can create a document for each day, and have an array of messages inside that document.
You can create a document with the _id’s of all the recent posts/messages/whatever, and then start a new subscription for each document using the _id. When the array of _id’s in that top document change, you can stop the unnecessary subscriptions and start the one’s that are missing.
Indeed - and it doesn’t have to go away. LiveQuery and LiveDocument are complementary. However, once you do hit your limits with LiveQuery, it would be nice to have a path for taming the beast.
I think it is good to discuss future developments, but to me it seems that the data layer toolbox in Meteor core and community proven associated packages already provides a rather diverse range of options as it is:
pub/sub as a very elegant and easy to use real time choice;
methods as simple RPCs for highly performant queries, also for using relational dbs;
Apollo/GraphQL for (at the moment non-real time) scalability, enhanced queries, relational dbs if need be;
RedisOplog for enhanced real time scalability;
Grapher for high peformance complex fetching;
classical REST as alternative to option 2.
In my humble opinion the main source that feeds the impression that Meteor might have performance issues in the data layer is that pub/sub as the default option is so easy to use that developers fail to account for alternatives and just use the real time option for everything. As load grows, issues arise and then the other options get somewhat tainted as well on an emotional level since nobody likes refactoring existing code (at least nobody I know ).
This is not to say that there is no room for improvement. Discussions like this here should be welcome. But I would slightly disagree with msavin’s comment in another thread that Meteor’s data layer needs fixing. Such wording for me seems a bit too strong.
Also, on a semi-related note, from the forums I get the impression that social media type apps (i.e. apps that generate very small profits per user, like ads, and instead focus on growing the sheer number of users to produce income) are somewhat over-represented in technical discussions when compared with the other end of the spectrum - the ‘dry and boring’ corporate apps that live in intranets and will never see a large spike in users and are therefore in many cases a very good fit for the existing pub/sub. Since the income generated per user in such cases is much higher than the ‘public apps’, the cost of extra resources server side does not really matter that much as long as it does not create any actual bottlenecks. From a strictly business point of view, such apps are not sexy but I would say critical for keeping any technical stack afloat in a long perspective.
I’m getting the sense that people really like oplog tailing and do not want it to change. The thing is, it doesn’t have to go away, it can still be used for small apps. However, when you get to big apps, it just doesn’t work.
Oplog tailing costs you a lot of money in terms of CPUs and imposes hard limits for scalability. It would be nice if it scaled perfectly, were cheap, fast, etc, but I don’t think that’s going to happen, and sometimes its best to work with what you have.
I think it’s really nice that MongoDB took notice of Meteor and launched some kind of support for real time functionality. By embracing it, Meteor can open itself up to use cases that go beyond those of hobbyists.
Also, despite Apollo, Meteor will always be a MongoDB stack at heart, and there’s nothing wrong with that. Meteor is a nice JS web stack, and MongoDB is a nice JS-ish web database. It’s got everything you need, it just needs to become more reasonable with the real time features.
Spot on @vooteles , I think you captured exactly what’s happening.
I also agree that the pub/sub API is a very elegant and easy to use and that for most apps (those with low volume) the current implementation will work well. For the extreme cases, where we’ve spikes from a ‘sexy’ public facing consumer app, ideally we’ve the MongoDB Change Stream API adapted for Meteor needs, but if that is not feasible, then I personally think RedisOplog is the way to go.
Regarding the Change Streams + Pub/Sub it seems similar to sending an event via web socket and then re-fetch using methods. As I mentioned earlier, personally I think the pub/sub is much easier to reason with for real-time updates, however (and I’m just thinking loud here, so please keep me honest) how about having “Reactive Methods” which simply get triggered based on a trigger defined on the backend and re-run at the client? it might be similar to one doc pub/sub and queries approach but cleaner, those reactive methods might act as a middle ground between full pub/sub and event/fetch or polling.
Pub/Sub keeps a cache of some data on the client and server. When that data changes, it diffs the old data and the new data on the server, and then sends that diff to the client. That way, the client has the same data as the database and server. (This is memory expensive.)
The problem is with how Meteor server finds out that the data in the database is changing. Originally, Meteor used to poll the database every 10 seconds and compare the query. That didn’t work so well, and a community member (Arunoda) implemented oplog tailing, which MDG later implemented on their own as LiveQuery.
Live Query watches all of MongoDB’s operational logs to see what changes, but has two big catches: first, its CPU expensive, second, every server must watch the log, and if it has to process too many items it’ll eventually give up or crash. This is what will force you off Meteor once you get real traction.
Change Streams basically say, “how about me (the database) just tells you (the server) when a change happens, and what that change is.” That way, Meteor doesn’t have to work so hard. However, it has its some trade offs, which is what the original post talks about.
I don’t think it’s about how many oplog entries it must process but how many queries it needs to keep track of. If your users subscribe to identical queries like Posts.find({ author: 'mpowaga' }); then each server will be listening for changes to this query. This could be improved by cursor deduplication on cluster level. But of course it won’t make much difference for queries like Posts.find({ author: Meteor.userId() });.
Listening to changes of set of documents as opposed to entire query is a good idea. I remember scoreboard demo and how it would display top N scores. However it’s not desired to keep track of the sorting in all cases. For example when you implement search functionality you don’t want to remove/add documents as they enter/leave matching set. You just want to subscribe to specific documents not the order of them.
It’s literally about how many oplog entries the servers must process.
Try to insert 10,000 documents at once into your database and Meteor will choke. (It should take much less than that, but that’s a number that ought to do the trick.)
That’s where the bottleneck lies. If its just about managing publications, it wouldn’t be such an issue because you can add more servers. However, each server must watch the logs, and eventually it can’t keep up with it.
Is it typical use case? I think it should handle steady write rate.
It’s O(n*q) where n is number of oplog entries and q is number of live queries. You can’t always reduce amount of writes but you can reduce number of queries especially when you run identical queries across the cluster which is the case for most Meteor apps.
There are much wiser questions to ask. And you could have been nicer here and quoted the entire sentence, which indicates it likely takes much less.
I have heard of just 800 inserts stopping Meteor servers, which can happen if just one user sync’s their address book or something to your service, or uploads a CSV, etc.
Let’s go back to the basics: Each insert/update/remove is posted on the oplog. An application with a couple of thousand concurrent users can easily push Meteor over that number.
There’s also a lot of other things happening on that Meteor server too. But, even if those logs wouldn’t crash your app, what good is it more than 50% of your processing power is being consumed by oplog tailing? Your app gets expensive to run, and that eats into your margins.
It would be interesting to see what subscriptions were active at that time. My point is that the biggest dependency is live queries you run than number of writes. For example when you don’t listen for any live query, Meteor can skip every oplog entry which is almost free. If you subscribe to documents by id then you perform equality check on document.id and oplog-entry.id which is fast. But when you have more complex query which is sorted then it will be very slow.
I hear you - but the ultimate goal is efficient pub/sub and efficient ways of updating servers of changes to data. The former is done well, we’re here to talk about the latter.
I think Apollo’s new setup since 2.0 will make subscriptions much easier. There is still some manual stuff (I think) as you’re writing all the logic on the server for what to do.
I’m chatting with @diaconutheodor right now, and he brought this quote from MDG to my attention:
“Including id in queries can also help improve scaling when using oplog tailing. In general, running queries on id will be faster and have more predictable performance. If it is possible to rework your schema so that _ids are used more extensively, especially on hot collections, you may able to get orders of magnitude improvement in performance.”
Subscriptions by _id + Change Streams may be the only realistic way to scale Meteor. I’m happy to make the adjustment if that means greater scale, performance and cost.
And for clarification - this does not mean it takes the pressure off the oplog tailing problem. It simply means the pub/sub mechanism can operate more efficiently. The oplog thing has to be deal with separately.
Between MDG’s statement that pub/sub works best on single document subscriptions, and Change Streams being perfect for this use case, I think we finally got the solution.
But MDG statement saying you’re better off using _ids when running queries, it’s not saying subscribe to single document if I understood it correctly. I’m still not clear what single doc subscription really means, it seems more restrictive then the pub/sub implementation we used to. Am I missing something here?
Currently, Meteor allows us to write reactive database queries by tailing MongoDB’s operation log. However, MongoDB recently announced a feature called Change Streams which paves the way for a more scalable but limited solution.
).