Application freezes with reactive vars

help
1

Hello,

I’ve been using reactive variables when my collection sizes was max size around 700 items or so but now, my collection holds more than 10,000 items.

I display the collection size on the client dashboard and it basically freezes the template until all the items are accounted for and then you can scroll and move around.

I’m doing all of this on the client site. Here’s an example of one of my function on the client site:

Template.totalDevice.helpers({ totalDevice: function() { var totalDevice = 0; Tweets.find({ tDevice: 1}).map(function(doc){ totalDevice += 1; setTotalDevice(totalDevice); }); return getTotalDevice(); } });

I have several helper functions like this going on the client side and they’re all reactive.

How can I make it so that the template doesn’t freeze every time the template is getting data?

Thank you in advance for your help! I really appreciate it.

2

So, you’re maintaining a running count of tweets vs device and need to keep this updated “for ever”?

At client start-up you make two actions:

  1. A method call to the server to aggregate and return results by device for tweets older than “now”. Using the aggregation pipeline for this is really fast if you’ve got your collection suitably indexed. You use the returned data to pre-seed the result sets.
  2. A subscription to the collection for tweets at or newer than “now”. You can use your helper to add to the original, seeded aggregates.

Your helper is then working with a much smaller set of data. You could also periodically refresh the “now” point to start again (for example when the cursor from the subscription is returning more than 100 results.

3

Hey Rob,

Thanks for your response. So, I changed the code up a bit and instead of going through the entire collection on the client side, I make a call to the server to give me the count for all device fields that are 1, so I used:
Tweets.find({} , { fields: {tDevice: 1} }).count(); and return that to the client. This has made things slightly faster but the issue is still there with the freezing.

How do I index the collection like you mentioned? Also, could you show an example of how to aggregate the collection for tweets older than now? Does it matter whether the tweets are being inserted into the collection very fast or not? On the server side, approximately 500 tweets are being inserted in less than 7-8 seconds into the collection.

4

So, generally speaking, it’s a good idea to index any fields you search on or sort by in a find (this includes fields where you want to find where createdAt < now (for example). In this case I would definitely add an index to createdAt. In your server-side Meteor.startup add a line like Tweets._ensureIndex({createdAt: 1});. Alternatively, you could do this through the Mongo shell. Once defined, it remains.

I need more information about your data and how you expect data to be aggregated. However, it’s not difficult to build an aggregation pipeline - take a look at this good introduction from the MongoDB docs.

Not particularly. As long as you are consistent in how you identify documents to be included in the aggregation (excluded from the publication) and included in the publication (excluded from the aggregation), it won’t matter how fast the documents are being inserted.

That’s not especially fast for MongoDB - but it could be an issue for your clients if you’re pushing that volume of data down to each connected client. You definitely need to understand your data flows and where/how they can be optimised. You could look at incremental aggregation techniques to speed up historical queries. This is where you have (a) separate collection(s) holding aggregated data. These will tend to be much smaller than the raw collection(s) and can be queried much more efficiently. You periodically update these aggregates (and maybe remove raw data from the source collection(s)).

5

Thank you so much. I will definitely try this. I have one more question: let’s say I make a call to a Meteor method from my client side to retrieve the count() of a collection in real time. Would I need to put the Meteor.call in an interval on the client side, so it can keep getting an updated number or can I simply put a Meteor.call and expect a response every time the value changes. I tried it with the setInterval and set a 10 second interval but you can see the data on the page reload every 10 seconds and it gets a bit annoying to look at.

6

Meteor methods do not provide any reactivity. If you want to use a method, using a setInterval is an option. Alternatively, you could take a look at the tmeasday:publish-counts package, but it’s not really suitable for the sort of activity you’re seeing.

Your best option may be to cache the running counts on the server as they arrive and use the publish api (added, changed, removed, ready) to publish updates at, say, 1 second intervals. Here’s a basic example of a roll-your-own publication (it publishes a REST endpoint, but you will get the idea):

7

Thanks Rob. I really appreciate all the help. I just made a setInterval function on the client side that makes the calls to the meteor methods every 10 seconds or so to update the data! This works out so far. I’m going to test everything once I convert all my functions to use aggregate, instead of Tweets.find({})

1 Like