I have a Meteor subscription that only needs to return 13 documents, and it is taking 5+ minutes for it to even hit my (local) server. For reference, there is another Meteor subscription that runs just before this one that returns 5000+ documents, and it only takes a few seconds. The subscription works and eventually the subscription is ready, but it takes 5+ minutes. I am baffled as to why that is. Does anybody have any idea what would cause this? Autopublish is off. Here is the code in question:
wip.forEach((w) => {
Meteor.subscribe('skuOne', w.skuID, {
onReady:() => {
Meteor.subscribe('custOne', w.custID, {
onReady:() => {
count++
if (count === wLength){
// Do stuff
}
}
});
},
onStop:() => {
console.log('failed');
}
});
});
1 Like
Here is what my publish functions look like:
Meteor.publish('skuOne', (id) => {
console.log('sku', id);
return Skus.find({'_id': id});
})
Meteor.publish('custOne', (id) => {
console.log('cust', id)
return Customers.find({'_id': id});
})
When I say it takes 5+ minutes, it takes 5+ minutes before even the console logs fire.
First of all this is highly inefficient way to do subscriptions. You are creating new cursor for each item and then potentially multiple cursors for one customer. So first step would be to get all the ids into an array and send that array in one subscription and you could do the same with the customer ids as well. This way you are creating only two subscriptions instead of subscription for each item and customer.
You might even want to consider publish composite if you want to reduce this to just one publication:
https://atmospherejs.com/reywood/publish-composite
2 Likes
As @storyteller rightly said, get the ids into arrays, and then you can manage the entire subscription (this one even without publish composite) in one go like this:
Meteor.publish('skuAndCust', (skuIds, customerIds) => [
Skus.find({ _id: { $in: skuIds } }),
Customers.find({ _id: { $in: customerIds } }),
])
3 Likes
Thank you for the tips. I have adjusted my code to make it more efficient. However, it is still taking upwards of 5+ minutes before my subscribe request hits the backend, which is my localhost. This is just after another subscription which subscribes to 5000+ documents. Here is my new code for reference:
(front-end)
let wipSku = wip.map((w) => w.skuID);
console.log(wipSku);
let wipCust = wip.map((w) => w.custID);
console.log(wipCust);
Meteor.subscribe('skusCusts', wipSku, wipCust, {
onReady:() => {
console.log('subscriptions loaded');
this.setState({
...this.state,
loaded: true
});
}
})
(back-end)
Meteor.publish('skusCusts', (skuIDs, custIDs) => [
Skus.find({_id: {$in: skuIDs}}),
Customers.find({_id: {$in: custIDs}})
])
I get no errors whatsoever and it does work eventually, but only after waiting for several minutes.
Do you use MongoDB Atlas by any chance? If so, some people have experienced extremely poor performance on certain tiers from time to time. Can it not be the reason? E.g. could you please add a temporary Meteor method that fetches exactly the same documents as the ones that you subscribe? How long does that take until that data arrives on the server, and how long until it eventually gets to the client?
1 Like
Not using Mongodb Atlas. This is a local mongo instance that the application is connecting to. I took your advice and added a test method to fetch the same exact documents. Once again, it takes several minutes for it to even hit the Meteor backend, which is also on my local machine, and read the console logs. Here is the code:
Front:
let wipSku = wip.map((w) => w.skuID);
console.log(wipSku);
let wipCust = wip.map((w) => w.custID);
console.log(wipCust);
Meteor.call('test', wipSku, wipCust, (err, res) => {
if (err){
console.log(err);
} else {
console.log(res);
}
})
Back:
Meteor.methods({
'test':(sku, cust) => {
console.log('test');
let skus = Skus.find({_id: {$in: skuIDs}}).fetch();
let custs = Customers.find({_id: {$in: custIDs}}).fetch();
return [skus, custs];
}
})
And as mentioned previously, just before this method/subscription runs, another subscription returning over 5,800 documents runs perfectly fine and it only takes a couple seconds.
If you open up chrome dev tools and refresh the page, then navigate to the network tab and filter by WS you’ll see the websocket. If you click on this you’ll see the DDP messages being sent.
When you’ve done this, trigger whatever functionality causes your subs (both the 5800 doc one and the slow one) take a look at the timestamps of the messages - then take a look at the timestamp of the log where you received the subscription. It could be that you’re stuck waiting for all the 5800 docs you requested to arrive, or something similar. In any case, comparing the timestamps of you issuing the call, the DDP channel showing the call and the server receiving the call will be useful
1 Like
I understand that the test method to fetch the same exact documents is also taking several minutes, which is good, at least we know that the poor performance is not related to publish/subscribe.
Would you mind checking the document sizes? If, for any reason, maybe unintended, they are really large, that would explain this.
1 Like
The 5800+ doc one is 25.3kB in size and is already completed before the problematic one initiates. So it is not a matter of waiting for the big one to complete as the big one is completed within a couple of seconds. The smaller one is 448 bytes. It should be noted that there is absolutely no activity on the Network tab when the subscription is supposed to go off. It waits several minutes, then fires off the subscription and gets a response within 104ms. So the problem appears to be with the subscription and not the publish. I have put console logs before and after the subscription which are both going off instantly. For whatever reason, Meteor is waiting several minutes to fire this off:
console.log(wipCust);
Meteor.subscribe('skusCusts', wipSku, wipCust, {
onReady:() => {
console.log('subscriptions loaded');
this.setState({
...this.state,
loaded: true
});
}
})
console.log('ok');
Note that both console logs go off instantly.
What about your MONGO_OPLOG_URL? Is that set correctly?
1 Like
To help you debug, install the Meteor devtools and see what’s happening
1 Like
don’t use fat arrow here
Meteor.publish('skusCusts', function (skuIDs, custIDs) {
return [
Skus.find({_id: {$in: skuIDs}}),
Customers.find({_id: {$in: custIDs}})
]
})
2 Likes
Nice catch, @crapthings . Something to add in the documentation as a warning
1 Like
What’s wrong with the arrow function here?
1 Like
Just checked the documentation and it is explicitly mentioned there about the explicit binding to the publish handler
Function called on the server each time a client subscribes. Inside the function, this is the publish handler object, described below.
This is easy to miss and maybe an explicit warning should be added.
1 Like
Using an arrow function is indeed a potential source of error, especially if someone isn’t aware of its differences with traditional functions with regards to the binding of this. Adding a warning to the documentation can’t hurt.
Other than that, in the scope of this discussion it makes no difference if the arrow function is used, so it’s not like “don’t use an arrow function here and this problem goes away”.
1 Like
I do not use the MONGO_OPLOG_URL, though I’m not sure how that would make a difference as the issue is that my method/subscription isn’t being fired from the client at all for several minutes.
I did try changing the arrow function to a regular function to no avail.
I will have to sit down and figure this out with the Meteor devtools when I have a few hours this weekend. Thank you for all of your replies.
Please check if any of your meteor methods is blocking the request. The method calls from the client are blocking by default, unless you use async methods or this.unblock. It is highly likely that such a call that is being executed before your sub is taking the aforesaid 5 mins.
On a side note, I would discourage you to subscribe 1000s or even >100 documents. You will soon start seeing performance bottlenecks. On the UI you would be showing only a limited number of data points, and I suggest you to send only that limited set from server. Use server side pagination.
1 Like
You’re so right: a Meteor method invoked first and taking long can hold up a subscription from even beginning to deliver data, if the method was not unblocked one way or another.
On the other hand, what does a method do that reliably doesn’t return for 5 minutes? It must be a hell of an operation!