Faster loading a big set of data


#1

Hello everyone! In my app, I have a template where I load a big set of conversations. I subscribe to set of data in onCreated callback:

Template.chat.onCreated(function onCreated() {
  this.subscribe('conversations');
});

And here is my publication:

Meteor.publish('conversations', function conversations(options = {}) {
  const optionsToUse = _.pick(options, ['limit', 'skip']);

  optionsToUse.sort = { date: -1 };

  Meteor.publishWithRelations({
    handle: this,
    collection: Meteor.participants,
    options: optionsToUse,
    filter: { userId: this.userId, deleted: { $exists: false } },
    mappings: [{
      key: 'conversationId',
      collection: Meteor.conversations,
      mappings: [{
        key: 'conversationId',
        collection: Meteor.participants,
        filter: { userId: { $ne: this.userId } },
        reverse: true,
        mappings: [{
          key: 'userId',
          collection: Meteor.users,
          options: { fields: { username: true, avatarId: true } },
        }],
      }, {
        reverse: true,
        key: 'conversationId',
        collection: Meteor.messages,
        options: { limit: 1, sort: { date: -1 } },
      }],
    }],
  });
});

And now, every time I open this template I must wait around 1 minute to load (In production, there are to much messages).

What I want? I want faster loading. I tried meteorhacks:fast-render, percolate:paginated-subscription. But It do not works for me. I want something like this: I open template, after short loading see first 10(for example) dialogs and then, when I already see some dialogs, other a loading in background. What can I do for it?


#2

Why not do something like:

Template.chat.onCreated(function () {
  let self = this;
  self.autorun(function () {
    let initSub = self.subscribe('conversations', 10);
    if (initSub.ready()) {
      self.subscribe('conversations');
    }
  });
});

Meteor.publish('conversations', function(limit) {
  let options;
  if (limit) {
    options.limit = limit;
  }
  // Your return code
});

Sub to the first ten, then when that pub comes back, resub to the larger set?


#3

I would also take a look at your indexes. It looks like you need to at least index all your relationship fields, filter fields and sort fields. So, the following:

  • participants: userId, deleted, conversationId
  • conversations: userId
  • messages: conversationId, date

Note: I haven’t used this package, so I may have got some of those wrong.


#4

I have Meteor.publishWithRelations function for publishing. I need to publish related data too(


#5

Indexes is Mongo Ids


#6

Mongo _id is a default index. However, you can (and mostly should) add extra indexes for performance.

https://docs.mongodb.com/v3.2/indexes/


#7

I need custom indexes for solving my problem?


#8

Yes


#9

And, how it will help me?


#10

Well here’s a simple optimisation. You are performing a sort by date with limit to get the most recent. If date is not indexed, then every document in the collection has to be fetched and sorted before the limit can be executed. If date is indexed it’s already sorted, which means the most recent document can be fetched in one read (actually one index read + one collection read). If you have 1,000,000 documents in your collection that’s way more than a million times faster (the sorting of those documents into date order will dominate the retrieval time).

Similar arguments apply where you are relying on filtering and “foreign keys”.


#11

It appears that you are using my socialize:messaging package. This thread has cause me to evaluate the indexes on the collections and I’ve found that I actually missed 2 very important indexes for this to run efficiently. I’ll publish a new version in a minute or 2 and things should run a little better…

That being said, you should definitely still paginate your data in some way. Even at it’s most optimized this can be an expensive publication.

Edit
v0.5.3 published.


#12

I use socialize: user-model package


#13

can you paste the output of meteor list?


#15

Package on use:
‘blaze-html-templates’,
‘check’,
‘ecmascript’,
‘jquery’,
‘less’,
‘reywood:publish-composite@1.4.2’,
‘aldeed:simple-schema@1.5.3’,
‘socialize:base-model@0.4.0’,
‘socialize:user-model@0.1.7’,
‘socialize:user-presence@0.3.4’,
‘socialize:server-time@0.1.2’,


#16

Going to leave this here as I think it could help: https://www.youtube.com/watch?v=z9-9PCaEpbU


#17

LOL, so you are using base-model, user-model, user-presence and server-time (all the dependencies of messaging) but not messaging itself?


#18

We use modified version of your package. I`ll try to add 0.5.3 changelog to our custom package


#19

ahh, that makes more sense then :slight_smile:


#20

So, what changes I need to add, can you help me with that?


#21