Meteor loading the whole collection


#1

I am trying to list just the top 100 items in a collection with a couple of where clauses based on the client side settings. Default order is by the timestamp (newest first). The database is our log file, so it’s generated outside of Meteor.

But when I open the page, it seems to page through the whole collection starting from oldest to newest. It seems to keep just the top 100, but scrolls through the whole collection. Often it seems like it crashes or something, then starts over again. It gets SUPER slow loading that last few items. Right now it has around 10k documents. What do I have wrong?

My server code:

  Meteor.publish("logs", function () {
    var items = Logs.find({}, {sort: {timestamp: -1}}).fetch(); //I get an error if I return this
    return Logs.find();
  });

Client code in template body helper:

logs: function () {
  var level = Session.get("level");
  var limit = Session.get("limit") || 100;
  var levelArray = ["DEBUG", "INFO", "WARN", "ERROR", "FATAL"];
  var index = levelArray.indexOf(level);
  if (index > 0) {
    levelArray.splice(0,index);
  }
  var isReactive = Session.get("isReactive");
  var items = Logs.find({ level: { $in: levelArray }}, {sort: {timestamp: -1}}).fetch();
  items = limit % 1 === 0 ? items.splice(0,limit) : items;
  return items;
}

#2

You need to limit the results when you publish the Collection from the server, you also don’t need to .fetch() when you publish… also make sure you don’t have autopublish package still.

A quick google search, this has an answer with limit in the publish.


#3

I mean, can you even use fetch? I think you can only return a cursor or an array of cursors


#4

I don’t remember how I came to use fetch. On the server, no it errors. (I think that was commented out before.)

I updated my publish to return:

      return Logs.find({ level: { $in: levelArray }}, {sort: {timestamp: -1}, limit:limit});

But now it’s not reactive at all (doesn’t update as records are added by external service. And how do I pass variables to it?

Also, is the publish supposed to always be in the Meteor.startup() function or just on the server?


#5

This blog post covers exactly how to subscribe to a reactive publication that depends on limit.


#6

Thanks, that looks pretty helpful. I’ll see if I can get it working. Much appreciated!


#7

I ended up using the MeteorPad code from the blog post you showed. It is very similar to what I wanted to build. I got it to work with my database, and you could load more documents on demand. However, when new documents are added (external source), they don’t show up. I have to actually refresh the page. I changed the query to return the newest records as such:

return Posts.find({}, {sort: {timestamp: -1}},  {limit: limit});

Otherwise, it is basically the same code. Thoughts?


#8

I’m still stuck on this. If you have any thoughts, I’d greatly appreciate it.

I am sorting by the newest documents first. When newer documents are added straight to the database, they don’t show up and push the other documents down, off the page like I was hoping. I’m guessing nothing triggers the publication on the server to update the client.


#9

Hard to tell without seeing any code. Could you post how you are doing to subscribe to the data?


#10

This is the client subscription code:

  instance.logs = function() { 
    //return Logs.find({}, {sort: {timestamp: -1}}, {limit: instance.loaded.get()}); //using this causes the entire collection to show, but does update the UI properly
    return Logs.find({}, {limit: instance.loaded.get()});
}

As my comment states, the first return DOES update the UI, but displays all the documents. The second one (from the post you suggested) only displays the first 5.

However, in both scenarios, the entire collection is still pulled into the client. When I do a Logs.find().count() in the console, I get the same count as from the mongo instance. I would think that wouldn’t work if I have 1000’s of records eventually in there.


#11

You are misunderstanding how publications and subscriptions work. You publish a subset of your data and subscribe to it, refining the search when you actually do the find. But even when you limit the data displayed, you are querying over the subset was sent to you by the server.

If you only want to display the first five, pass the limit option as parameter inside your publication. Then update this parameter reactively inside a Tracker.autorun() function. This way you will resubscribe with different parameters when the reactive variable changes and only the right amount of data will sent to you.


#12

I think I do get the pub/sub model. My publish has a limit parameter passed in.

The difference is nothing gets updated on the client (only when more documents want to be shown). When new documents get added, they don’t show up. That’s what I’m trying to do.

Here is the full js file:

Logs = new Mongo.Collection(“logs”);

if (Meteor.isClient) {
  // counter starts at 0
Template.logs.created = function () {

  // 1. Initialization
  
  var instance = this;

  // initialize the reactive variables
  instance.loaded = new ReactiveVar(0);
  instance.limit = new ReactiveVar(5);
  
  // 2. Autorun
  
  // will re-run when the "limit" reactive variables changes
  this.autorun(function () {

    // get the limit
    var limit = instance.limit.get();

    console.log("Asking for "+limit+" logs…")
    
    // subscribe to the logs publication
    var subscription = instance.subscribe('logs', limit);

    // if subscription is ready, set limit to newLimit
    if (subscription.ready()) {
      console.log("> Received "+limit+" logs. \n\n")
      instance.loaded.set(limit);
    } else {
      console.log("> Subscription is not ready yet. \n\n");
    }
  });
  
  // 3. Cursor
  
  instance.logs = function() { 
    //return Logs.find({}, {sort: {timestamp: -1}}, {limit: instance.loaded.get()}); using this causes the entire collection to show
    console.log("logs on instance: ", Logs.find({}, {limit: instance.loaded.get()}).count());
    return Logs.find({}, {limit: instance.loaded.get()});
  }
  
};

Template.logs.helpers({
  // the logs cursor
  logs: function () {
    return Template.instance().logs();
  },
  // are there more logs to show?
  hasMorePosts: function () {
    return Template.instance().logs().count() >= Template.instance().limit.get();
  }
});

Template.logs.events({
  'click .load-more': function (event, instance) {
    event.preventDefault();
    
    // get current value for limit, i.e. how many logs are currently displayed
    var limit = instance.limit.get();
    
    // increase limit by 5 and update it
    limit += 5;
    instance.limit.set(limit)
  }
});
}


if (Meteor.isServer) {
Meteor.startup(function () {
  if (Logs.find().count() === 0) {
    for (i = 0; i <= 50; i++) {
      Logs.insert({
        title: Fake.sentence(6),
        body: Fake.paragraph(3)
      });
    }
  }
});

// publish logs
Meteor.publish('logs', function(limit) {
  Meteor._sleepForMs(200);
  return Logs.find({}, {sort: {timestamp: -1}},  {limit: limit});
});
}

#13

Well, this is expected as return Logs.find({}, {limit: instance.loaded.get()}); is not sorting. Last entries will be at the “bottom” and will not be shown once you have a limit.

You don’t seem to have a timestamp field (I’m not sure if Mongo provide a default value for it, but my guess is no) so apparently using return Logs.find({}, {sort: {timestamp: -1}}, {limit: instance.loaded.get()}); will return values in any order (ran some tests here and this happened to me).

What if you do like this?

Logs.insert({
  title: Fake.sentence(6),
  body: Fake.paragraph(3),
  timestamp: new Date()
});

#14

Thanks for the help. I ended up purchasing the Discover Meteor book and using the Microscope project to build my base from. The project has some similarities to what I need.

Thanks for the help!