Good point - I updated the post to mention subscriptions by _id.
2 Likes
Agreed. Could you explain what a single document subscription means a little more? @msavin For example, right now the majority of my queries are based on fields IE return all students in grade 10. db.students.find({‘grade’:10}) and my subscription would be based off that. However, that may return hundreds of student documents from the DB. How would the single doc subscription work based on that example? Would I have a single document that contained all my student objects and subscribe specifically to that _id? Does that affect reactivity in any sort of fashion?
Btw, really appreciate all your effort/time on this matter. Been reading github responses and greatly appreciate the things you outlined in this thread. Scaling my Meteor app is coming up very soon so this is a topic I care about a lot.
2 Likes
Getting students by a grade - since that returns hundreds if not thousands of documents - may not be a good use case for Pub/Sub. Instead, you might prefer to feed that data through a Method or something like Grapher. It should be much faster too.
Here’s a quick example of where this could work though:
// on the server
Meteor.publish("classRooms", function (list) {
return Rooms.find({
_id : {
$in : list
}
});
})
// on the client
// (might need some reactivity magic)
Meteor.subscribe("classRooms", Meteor.user().classRoomList)
In that example, instead of giving pub/sub a query, you tell it exactly which documents you need. It can do its job a lot more efficiently because it doesn’t have to check if new/updated documents match it’s query, it only needs to keep an eye on the _id’s.
Change Streams fit this use case perfectly because they are intended to watch documents and not queries. Plus, I think you can trigger updates only when specific fields are changed, such as classRoomList in a user’s profiles.
It would have the same result as the query below, but it requires us to change how we think about things, but it would be a lot more efficient.
Meteor.publish("classRooms", function () {
return Rooms.find({
students: Meteor.userId()
});
})
The reason I like single document subscriptions - where you subscribe to just one document by its _id - is that it has very high re-usability across different Meteor clients. Meteor only needs to create one subscription in the server(s).
In that case, maybe you can get better performance by starting a subscription for each item in Meteor.user().classRoomList, and stopping/starting subscriptions as items in that array change.
2 Likes
I have no idea, how it will be in Apollo 2, but now Apollo PubSub doesn’t scale to multiple servers…
From http://dev.apollodata.com/tools/graphql-subscriptions/external-pubsub.html
This is not suitable for running in a serious production app, because there
is no way to share subscriptions and publishes across many running servers.As a hobbyist and somebody whose only Meteor app relies almost entirely on syncing different clients’ UI state based on observeChanges from single document subscriptions, I’m pretty sure I’m excited about this way forward
1 Like
yeah really like this model actually, a sort of control document could give us far more fine-grained control.
From recent experience, fetching data over methods is super awesome, - would be great to add some convenience functions though for working with minimongo when doing so, in terms of controlling the client side collection.
1 Like
I suspect for many, it’s not so much that we like oplog tailing as we’ve written quite a bit of code based on pub/sub, and would prefer not to have to do a major refactoring of our whole app if we could avoid it.
I’d find it interesting if there were an option to have pubs that used Mongo Change Streams rather than oplog tailing. It would loose noticing when records are added to a query, so not as responsive as we currently enjoy. But the initial query gets all the records in question, so could get all the ids in question, and do change streams on them. Would result in more scalability with some loss in functionality, but not needing to refactor our code much.
As for the efficiency of getting data to the client, with this model, I think we could speed things up a bit, and reduce memory constraints on the server. If Mongo change streams are telling the server what’s changed, it could use DDP to send the changes to the client without storing what the client used to have in server memory. Beyond that, if methods are still faster than pubs, is there any reason we couldn’t do similar data transmission for pubs too? This sort of model could leverage the speed and scalability that you’re talking about, while keeping data in minimongo, enabling us to not need to refactor as much code. Somethign worth considering I think.
2 Likes
I want to watch over just the inserts on a collection matching a particular field, and push the newly inserted docs to all subscribed clients.
Change Streams will efficiently push the matching docs as they are inserted on the fly, but I am not sure if Oplog tailing by Meteor would give the same efficiency or not. As I understand it, with every update on the Oplog i.e. with any mutation on my whole DB cluster (including the collections not part of my subscription?), Meteor would check if it’s changing my published cursor, is it so? So, is it like Change Streams is pushing just one-way from Mongo to Meteor but Oplog Tailing is like a round trip and hence less efficient.
@msavin, does a publication that uses an aggregate query to join data with $lookup still count as a valid single document publication?
Something like:
// pseudo / untested
Meteor.publish('comments', blogId =>
db.blogs.aggregate([
{ $match: { _id: blogId } },
{
$lookup: {
from: 'comments',
localField: '_id',
foreignField: 'blogId',
as: 'comments',
},
},
])
);
If it does; then we could still divide our data into multiple documents / collections in mongo. But keep a more simple and performing publication.
It would also make it a lot easier to refactor existing code, as it doesn’t require any database migrations or changes in insert / update statements.
1 Like
@msavin, anymore word on LiveDocument integration with Meteor? Have you seen any PRs on this? Does Meteor 1.7+ change things on this front at all?
Thanks!
Nothing new here. I’m really hoping Meteor’s MongoDB package gets an update soon - a lot of new things have come out lately.