Publish is slow - why?

help
1

Hello everybody!

I have a simple publish function that takes a long time to run. I’m not sure why. The count goes instantly, and when I run the query in mongodb shell, it runs instantly. But in my publish/subscribe, it takes about 10 seconds for the documents to go across the websocket and show up in my application. When I had only a few items in the collection, it was instant. But now with 100k dummy records, it is slow.

Counts.publish(this, 'entitiesCount', Entities.find({}), { noReady: true });
return Entities.find({}, {
    fields:{firstName: 1}, 
    skip:0,limit:10,sort:{updatedAt:-1}
});

The count is published instantly - no issue with that. I also have an index on updatedAt so no issue there. Further if I run this query in the shell, results come back instantly.

db.getCollection('entities').find({},{firstName:1}).limit(10).sort({updatedAt:-1})

Why is it so slow in my publish function?

2

100k realtime data ?

disable mergebox
collection reactive: false

3

Just pointing out you’re doing a Find All (find({}))

This could be the reason why. It’s looking through ALL your data, which is 100k.

Do you know why Google has Pagination? This exact reason. But it also caches the data. You’ll need to look at this concept closely. Cheers,

4

Oh my! Stop using reactive pub/sub when its not so important - use methds to fetch paginated data like

const PAGE_SIZE = 100;

Meteor.methods({
  getItems(page) {
    check(page, Number);
    const skip = Math.max(page - 1, 0) * PAGE_SIZE;
    const limit = PAGE_SIZE;
    const sort = {createdAt: -1};

    const cursor = Items.find({}, {skip, limit, sort});
    return {count: cursor.count(), data: cursor.fetch()};
});
5

Increase ddp rate limiter, then try to add index to updatedAt :slight_smile:

6

Ok guys, thanks for all the feedback!

I’m kind of bummed about this because one of the main reasons for using meteor was reactivity on everything! Would somebody please explain what it is about reactivity that makes it so slow vs the raw query which is fast?

7

@SkyRooms, I am running pagination on this. But it works instantly if I put the query into robomongo. It’s just something about meteor’s publish that is making it slow.

8

What is Entities.find({}) doing in the parameter list for this method?

Have you ensured updatedAt is an index?

9

Do you have a typo? This code does not make sense:

Counts.publish(this, 'entitiesCount', Entities.find({}), { noReady: true });

There is no publish function on a collection?

10

@robfallows, @brucejo,

The counts.publish is from a Meteor package recommended on the Angular Meteor tutorial. All it does is publish the count of a query. It works instantly - the count is sent to the client without delay. The delay only occurs when sending the actual documents. @robfallows, I have an index on updatedAt. Also, when I run it in robomongo, it returns the documents in just milliseconds, so I know the query isn’t the issue.

11

@mrzafod, reactive pub sub was one of my biggest reasons for using meteor! I think it adds value to users, to see the list updated in real time.

12

Looking at the github page of publish-counts it says

Publish-counts is designed for counting a small number of documents around an order of 100. Due to the real-time capability, this package should not be used to count all documents in large datasets. Maybe some, but not all. Otherwise you will maximize your server’s CPU usage as each client connects.

try your publish function without invoking Counts.publish and see if the performance changes

13

Right, so then the count this guys doing should be in increments of 100, then push that data down to the client.

14

In your example you are getting 2 observers - first for Counts and second for publish handle. So your CPU/RAM usage growes up. Its to long to explain all the corners of reactive publish))

Anyway you could take a look for brillinant Kadira Academy to find more about Meteor performance

15

Ok, @jamgold wins the prize. Even though the count published instantly, and the main query results wouldn’t come back for 10 seconds, it still turns out the count publish was the issue. Disabled it and got results immediately! Now I’m going to have to find another pagination solution, but glad I know the root of the problem.

You guys are capital 8 gr8!

1 Like
16

Found the package!

https://atmospherejs.com/ros/publish-counts

I am using this with fastCount:true, and it works instantly again!

Thanks everyone for the help.

1 Like