Hello,
I want to understand why when I subscribe to 10,000 documents the subscription works slow (~10 s to load). I’m running it on a 3.2 ghz i7 with SSD. How can this be ?
I really think that the performance can be dramatically improved, but then again I don’t really understand the full flow of things.
Is there anyone that can give me some high-level guidelines into understanding how it works ?
I am willing to contribute 
The question is why do you want to subscribe to 10,000 documents at once, ie. why do you want to have all this data on the client? Also, are you doing it via one subscription, or are you using many?
Hi, ofcourse in a real scenario I limit it or use methods. I don’t need 10,000. However! The problem isn’t when a single person does that, but when 100 people subscribe to 100 different documents. I want to know why is the CPU so forced.
I really think this issue can be solved. Like the guys solved the 1.4.2 speed. Improving subscriptions speed can be the next step.
This is a different scenario. When one person subscribes to 10,000 documents, the single connection between user and server becomes a bottleneck. Meteor should handle 100 or more smaller subscriptions for different users just fine 
. Plus, you’re probably testing this on your local machine or in a VPS. Normally if you have a heavy used app you would use a dedicated MongoDB provider like mLab or Compose, which has much better DB performance.
I fully understand what you are saying. But realistically why should more than 100 documents be such a problem ? Just why ? We need to improve that.
Like I said, >100 documents is no problem for Meteor. In my app a single user subscribes hundreds of documents and it doesn’t break a sweat
I think we probably need to see the code you’re using in your pub/sub.
Just create 10,000 dummy elements and subscribe to it locally. They can have a single field called “name”. Tell me if it works flawless to you.
My concern is, why shouldn’t it work for 10,000 ?
You missed one important factor. In Meteor subscription is not only a “driver like” connection. It’s not a matter of 10.000 documents coping as well (into the minimongo). It’a matter of reactivnes. Once you establish a 10.000 documents subscription all of them have to be two-side “synchronized” - all the time. That’s what loads the connection and makes the difference. That’s the price for technology used.
What are the 2 sides you are talking about ?
mongodb - server side and minimongo - client side.
Client works on the local (browser) db copy only, based on minimongo library filled with ‘subscribe’ command.
Documents, available in the local (browser) minimongo, are limited by the subscription “limits”. The delivered collection (into the minimongo) can be then limited more by client side ‘find’ calls.
What a mess …
Hi diaconutheodor, and the others,
I think your problem comes from the fact that you don’t really understand the mechanisms of pub/sub.
For example, in MySQL or other database system, if you ask suddenly 10,000 documents (rows) it’s not going to be fast neither.
In meteor, in your publish you say : I want to put these data on the minimongo (client side / client computer) to be used when I subscribe them.
After I can be wrong about your understanding but with that :
I kind of guess that it misses you some information somewhere on this subject.
In any case, if you want to put for example 10,000 documents on the screen of the user, it’s gonna take a while (like you said around 10 sec.
And if you have 100 users doing that of course your server is going to get into some trouble depending on the power of it.
@pikiwix @paku
You don’t really understand my concern.
I am not putting 10,000 docs on the screen view. I just want that when I subscribe to around ~10,000 items I want it to be performant even if 10,000. I don’t do this, I know the limitations I keep a max of 20-30 per subscription.
My concern is not how to go around this right now, my concern is why 10,000 is so slow. When mongodb can deliver 10,000 objects < way under 500ms. Where is the bottleneck exactly ?
I made a test.
// both client and server
Messages = new Mongo.Collection('messages');
// server
if(Messages.find().count() === 0) {
for(let i = 0; i < 10000; i++) {
Messages.insert({
text: 'Message ' + i,
});
}
console.log('Inserted 10000 messages.');
}
Meteor.publish('messages', function() {
return Messages.find();
});
// client
Meteor.startup(() => {
const sub = Meteor.subscribe('messages');
const time = new Date().getTime();
Tracker.autorun(() => {
console.log(Messages.find().count(), new Date().getTime() - time);
if(sub.ready()) {
console.log(Messages.find().fetch());
}
});
});
Here is the console output:
main.js:123 0 3
main.js:123 1865 1246
main.js:123 2096 1293
main.js:123 5199 1872
main.js:123 10000 2367
main.js:126 [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object…]
So the 10 000 documents were steamed to the client, the total loading time was ~2 seconds and the first batch of documents was available in ~1 second.
As I wrote, price for reactivnes. You can not compare 10.000 rows one way transfer (sql statment) with reactive two way db-client connection.
If you want to fast copy those 10.000 rows just stop using publishing/subscribe method. Try to go with poor mongodb driver for nodes (if is there any) …
Also another difference between 100 users requesting 100 documents and 1 user requesting 10000 documents is that Meteor will reuse the pub if it can. So when you just have an query requesting all the documents in a collection and 100 users are requesting this pub, meteor will use the same pub for all 100 users.
For more information, read this article by the kadira team: https://kadira.io/academy/meteor-performance-101/content/improve-cpu-and-network-usage
I also don’t think MDG is planning on working on DDP or any of the Mongo reactivity stuff anymore. They’re focusing on Apollo and migrating everyone to GraphQL.
Hi, I am having the same problem for a subscription which only returns an only document and I don’t know why it is taking a long time to be ready.
Here is my publish endpoint:
const defaultAudioOfDevicePublication = new PublishEndpoint('device.monitoringButtons', function() {
ReactiveAggregate(this, Meteor.users, [
{
$match: { _id: this.userId }
},
{
$lookup: {
from: 'audios',
localField: 'profile.defaultAudio.idAudio',
foreignField: '_id',
as: 'audio'
}
},
{
$unwind: {
path: '$audio',
preserveNullAndEmptyArrays: true
}
},
{
$project: {
'profile.defaultAudio': 1,
'profile.simulacrum': 1,
'profile.siren': 1,
'audio': 1
}
}
], { warnings: false });
});
And in the client side I am using a simple ddp client:
let startTimeSub = performance.now();
try{
const clientSub = meteorClient.subscribe('device.monitoringButtons');
await clientSub.ready();
}catch(exception){
console.error('Error in subscription: ',exception);
}
console.log(`Time taken for subscription in client ${ meteorClient.authData.identifier }: `, performance.now() - startTimeSub);
I got the following test results (in milliseconds):
Time taken for subscription in client 1: 8976.605574004352 (eight seconds)
Time taken for subscription in client 1: 10307.169658001512 (ten seconds)
Time taken for subscription in client 1: 4713.364351000637 (four seconds)
Time taken for subscription in client 1: 144896.64724399894 (one hundred forty four seconds)
The only thing that occurs to me that is causing this is the collection of the publication which has about 1000 documents. However, I verified to put indexes on it in order to optimize the query, these are the indexes:
Note: After these tests, I used redis-oplog package and this improved much better (up to 250 ms in the subscription) but I had problems with login method from simple-ddp client, so, I decided to get back mongo.
I am using Meteor 1.11.1 and Mongo 4.4.0.
I wonder if anyone can help me with this issue using MongoDB?
Have you checked if the pipeline execution plan (explain) looks sensible?
You can add a debug callback to the options - something like:
],
{
warnings: false,
debug(plan) => {
console.log(JSON.stringify(plan, null, 2);
},
});
});