I am trying to list just the top 100 items in a collection with a couple of where clauses based on the client side settings. Default order is by the timestamp (newest first). The database is our log file, so it’s generated outside of Meteor.
But when I open the page, it seems to page through the whole collection starting from oldest to newest. It seems to keep just the top 100, but scrolls through the whole collection. Often it seems like it crashes or something, then starts over again. It gets SUPER slow loading that last few items. Right now it has around 10k documents. What do I have wrong?
My server code:
Meteor.publish("logs", function () {
var items = Logs.find({}, {sort: {timestamp: -1}}).fetch(); //I get an error if I return this
return Logs.find();
});
Client code in template body helper:
logs: function () {
var level = Session.get("level");
var limit = Session.get("limit") || 100;
var levelArray = ["DEBUG", "INFO", "WARN", "ERROR", "FATAL"];
var index = levelArray.indexOf(level);
if (index > 0) {
levelArray.splice(0,index);
}
var isReactive = Session.get("isReactive");
var items = Logs.find({ level: { $in: levelArray }}, {sort: {timestamp: -1}}).fetch();
items = limit % 1 === 0 ? items.splice(0,limit) : items;
return items;
}
You need to limit the results when you publish the Collection from the server, you also don’t need to .fetch() when you publish… also make sure you don’t have autopublish package still.
A quick google search, this has an answer with limit in the publish.
I ended up using the MeteorPad code from the blog post you showed. It is very similar to what I wanted to build. I got it to work with my database, and you could load more documents on demand. However, when new documents are added (external source), they don’t show up. I have to actually refresh the page. I changed the query to return the newest records as such:
I’m still stuck on this. If you have any thoughts, I’d greatly appreciate it.
I am sorting by the newest documents first. When newer documents are added straight to the database, they don’t show up and push the other documents down, off the page like I was hoping. I’m guessing nothing triggers the publication on the server to update the client.
instance.logs = function() {
//return Logs.find({}, {sort: {timestamp: -1}}, {limit: instance.loaded.get()}); //using this causes the entire collection to show, but does update the UI properly
return Logs.find({}, {limit: instance.loaded.get()});
}
As my comment states, the first return DOES update the UI, but displays all the documents. The second one (from the post you suggested) only displays the first 5.
However, in both scenarios, the entire collection is still pulled into the client. When I do a Logs.find().count() in the console, I get the same count as from the mongo instance. I would think that wouldn’t work if I have 1000’s of records eventually in there.
You are misunderstanding how publications and subscriptions work. You publish a subset of your data and subscribe to it, refining the search when you actually do the find. But even when you limit the data displayed, you are querying over the subset was sent to you by the server.
If you only want to display the first five, pass the limit option as parameter inside your publication. Then update this parameter reactively inside a Tracker.autorun() function. This way you will resubscribe with different parameters when the reactive variable changes and only the right amount of data will sent to you.
I think I do get the pub/sub model. My publish has a limit parameter passed in.
The difference is nothing gets updated on the client (only when more documents want to be shown). When new documents get added, they don’t show up. That’s what I’m trying to do.
Here is the full js file:
Logs = new Mongo.Collection(“logs”);
if (Meteor.isClient) {
// counter starts at 0
Template.logs.created = function () {
// 1. Initialization
var instance = this;
// initialize the reactive variables
instance.loaded = new ReactiveVar(0);
instance.limit = new ReactiveVar(5);
// 2. Autorun
// will re-run when the "limit" reactive variables changes
this.autorun(function () {
// get the limit
var limit = instance.limit.get();
console.log("Asking for "+limit+" logs…")
// subscribe to the logs publication
var subscription = instance.subscribe('logs', limit);
// if subscription is ready, set limit to newLimit
if (subscription.ready()) {
console.log("> Received "+limit+" logs. \n\n")
instance.loaded.set(limit);
} else {
console.log("> Subscription is not ready yet. \n\n");
}
});
// 3. Cursor
instance.logs = function() {
//return Logs.find({}, {sort: {timestamp: -1}}, {limit: instance.loaded.get()}); using this causes the entire collection to show
console.log("logs on instance: ", Logs.find({}, {limit: instance.loaded.get()}).count());
return Logs.find({}, {limit: instance.loaded.get()});
}
};
Template.logs.helpers({
// the logs cursor
logs: function () {
return Template.instance().logs();
},
// are there more logs to show?
hasMorePosts: function () {
return Template.instance().logs().count() >= Template.instance().limit.get();
}
});
Template.logs.events({
'click .load-more': function (event, instance) {
event.preventDefault();
// get current value for limit, i.e. how many logs are currently displayed
var limit = instance.limit.get();
// increase limit by 5 and update it
limit += 5;
instance.limit.set(limit)
}
});
}
if (Meteor.isServer) {
Meteor.startup(function () {
if (Logs.find().count() === 0) {
for (i = 0; i <= 50; i++) {
Logs.insert({
title: Fake.sentence(6),
body: Fake.paragraph(3)
});
}
}
});
// publish logs
Meteor.publish('logs', function(limit) {
Meteor._sleepForMs(200);
return Logs.find({}, {sort: {timestamp: -1}}, {limit: limit});
});
}
Well, this is expected as return Logs.find({}, {limit: instance.loaded.get()}); is not sorting. Last entries will be at the “bottom” and will not be shown once you have a limit.
You don’t seem to have a timestamp field (I’m not sure if Mongo provide a default value for it, but my guess is no) so apparently using return Logs.find({}, {sort: {timestamp: -1}}, {limit: instance.loaded.get()}); will return values in any order (ran some tests here and this happened to me).
What if you do like this?
Logs.insert({
title: Fake.sentence(6),
body: Fake.paragraph(3),
timestamp: new Date()
});
Thanks for the help. I ended up purchasing the Discover Meteor book and using the Microscope project to build my base from. The project has some similarities to what I need.