I was experimenting a bit and noticed that retrieving the same data via a method is MUCH faster than through a subscription. Around 50ms for the method, compared to 300ms until the subscription returns ready. Does anyone know the reason for this? Maybe it’s got something to do with subscriptions sending the initial data in small chunks?
A meteor method is just a call and response, whereas a subscription is basically setting up a data sync between the server and client, and then populating it. I am pretty sure at its core, subscriptions use methods too. The mongo & mini-mongo bit would add most that difference in latency. I am sure someone more experienced will be able to elaborate with a better explanation.
Ah, I had a look at what exactly a typical cursor publication does. Here’s the code:
Mongo.Collection._publishCursor = function (cursor, sub, collection) {
var observeHandle = cursor.observeChanges({
added: function (id, fields) {
sub.added(collection, id, fields);
},
changed: function (id, fields) {
sub.changed(collection, id, fields);
},
removed: function (id) {
sub.removed(collection, id);
}
});
So I guess it sends one ddp message for each document, compared to a method which only sends one message with all the documents. I guess that’s the reason why it’s slower? Seems like it’d be better if it could send the whole initial data in one message.
There is a price to pay for reactivity. If you do not need it, go for methods
1 Like
Most of the overhead is likely due to DDP. Unfortunately a DDP “added” message is sent for every record of a cursor that is published. I think things could be sped up a lot if there were a specification in the protocol for batching messages, that way if a subscription returns a cursor of 300 docs they could all be sent in one packet.
3 Likes
DDP just handles basic messages, not the details of pub/sub, added/changed etc. right? Instead of changing DDP maybe you could just make it so “added” supported an array of documents? If it could be 6x faster that would be pretty nice.
@ralof To me this seems like an unnecessary price though, just a matter of optimization. The real price is server load, but I don’t see why it should be slower to send data.
1 Like
If, for instance, “added” was sent as an array, where should the check be for just “your” element? I would have to loop through the whole changed array?
Hmm what do you mean? What’s “my” element?
the specific element that you have on display or want to watch for whatever reason.
That’s what collection.find() does, which returns the cursor observed by cursor.observeChanges in the code I posted.
Let’s say that the query you subscribe to returns 100 documents.
The “added” callback of observeChanges will run 100 times.
As it currently works, this will send 100 “added” messages through DDP.
What would be nice is that instead of sending a message right away, the “added” callback pushed the doc to an array, and once there are no more to add, that array gets sent as a single “added” message.
I can’t find where the code is that handles the recieved “added” message on the client though 
1 Like
@herteby If the changes you’re making using added/changed/removed look sort of like this (referring to the rooms tutorial in the Meteor docs): this.changed('counts', sidebar, { count });, then on the client I’m just doing import { Counts } from '/imports/api/counts/counts.js'; (in that file I’ve defined the collection with export const Counts = new Mongo.Collection('counts');). Normal client stuff like Counts.find().fetch() should work the same as if you returned a normal cursor-based publish.
To your other point about DDP, I’ve noticed this exact behavior with an application I’m currently working on. I’ve got ~500 documents in a collection and I’m using the added/changed/removed API to get a reactive count for the app’s sidebar (checking how many read, unread, and total items there are). Since the sidebar is persistent, it’s located in a parent React component relative to most other content, and I’ve noticed that it takes longer to load the count now that I have more documents. The problem is, this is now blocking every other child component from loading for a few seconds.
I’m currently not sure whether it’s worth going with a non-reactive static count at pageload or if it’s worth the extra overhead of DDP to have reactivity on the counts. The reactivity is a pretty core feature because our app is similar to email and knowing the read, unread, and total counts at any given time (especially after specific user actions) is important.
Not sure what you mean, but find() and observe on the client are reacting on changes by done the server. When you do an observe you are normally interested in the specific changes in the individual documents so if the server returns 100 documents in one batch you’d still need to loop through them if you want something to happen when a particular document are affected. And if you are using observe, well, that is normally what you are observing. If you do not need to observe, just don’t do it, find() is still reactive.
@alanx Yes but I’m talking about the step before that, the code that takes the DDP messages from the server and inserts the documents into the client collection. I think it’s ddp-client/livedata_connection.js but there might also be something in minimongo.
@ralof I’m talking about find() and observe in the publication on the server, when you create a publication like
Meteor.publish('asdasd', function(){
return Stuff.find()
})
The cursor returned by find() is passed to the _publishCursor function I posted, which when the subscription is initialized in my scenario sends 100 DDP messages to the client instead of potentially just 1.
“observe” on the client observes the local collection, not DDP messages. Changing the DDP messages between the client and server would not impact it.
2 Likes
But is using a method actually as easy as subscribing?
How could we get something like this to work in a React component?:
import { createContainer } from 'meteor/react-meteor-data';
export default createContainer((props) => {
let item
Meteor.call("getItem", {}, (error, result) => {
item = result
})
const container = {
item,
}
return container
}, Item)
1 Like
Call the method in componentDidMount and update local state in the callback. Even easier than pub/sub.
2 Likes
this seems like a huge win… i wonder what kinds of issues batching might come with, but i feel like this should be implemented
Oh I see now, I didn’t know that added/changed/removed were part of the DDP spec. I kind of thought you were free to design your own messages. That complicates things a bit, because I don’t know how willing MDG would be to change what’s supposed to be a spec for interoperability…
OK calling methods in componentDidMount and componentWillMount has worked when I tried it. The important thing to do was to call this.setState() to force update on the page after the data was retrieved by the method. It works when calling in the constructor too:
constructor(props) {
super(props);
this.state = {};
Meteor.call("getItem", {}, (error, result) => {
this.setState({ item: result })
})
}
The Meteor data container can then be empty:
import { createContainer } from 'meteor/react-meteor-data';
export default createContainer((props) => ({}) Item)
One thing @tab00 :
Be careful when using async calls in a React component. As per Facebook recommendations, asynchronous methods/activities should only take place within “componentDidMount”, after your DOM has been setup and you can actually update it as the data from your async call comes back to you.
Calling them in other places is an anti-pattern and one of the drawbacks is that you might end up getting the result (and triggering a state change) even before your component is fully rendered.
1 Like
Back in topic, I also found another way to get the benefit of reactivity without using Publications (in the traditional way).
I’m using this technique only when I just need to know when something has changed in the dataset I’m observing (still being reactive though) and the data I’m observing is not easily addressed with a simple find:
And example is when extracting data from multiple documents using Mongo aggregation.
In this case I use a method call to fetch data AND at the same time I provide a simple publication that I subscribe to in order to keep track of the “count” of the documents I’m observing…
When my subscription changes I know “something” has changed in my data set and I can trigger a new call to the method to fetch the aggregated data again. Wrapping this logic into a HOC makes it quite easy to use this approach when I need a not-super-fine-grained reactivity.
4 Likes