When do cached server queries clear

core
1

How long does a query on the server from a publish take to clear? For example if I subscribe for widgets and the find on the server limits the results to the widgets owned by the subscribing user, you will end up with a cached query for every user.

return Widget.find({userId: this.userId});

So when does it get cleared on the server? Is there a server setting for query cache timeout?

thanks in advance,
Rob

2

In Meteor, there’s not really a query cache. Instead, the publication sets up an observer on MongoDB’s oplog, so that mutations to the collection can be picked up for potentially passing on to the client.

As you’ve determined, each client subscribing to that particular publication will have its own observer, and that observer will persist for as long as the user is subscribed to that publication, or until they disconnect (e.g. close the browser tab).

So, if you want to stop an observer, and return memory and CPU, you can:

  • Stop the subscription.
    • if you’re using Blaze and template subscriptions, this is managed for you.
    • if you’re using Meteor.subscribe you need to manage this yourself through the stop method of the returned handle.
  • Stop the subscription in the publication through the this.stop() method.
  • Close the browser tab (disconnect).
3

Thank you for the response. It makes a lot of sense. It seems that our service on the server consumes more and more memory over time until we are forced to cycle it. I had been thinking that that it was caching, but since we use Meteor.subscribe and the observers should stop on disconnect, that is probably not the issue unless Meteor is leaking memory.

We will have to track it down some other way.

4

If you’re using Meteor.subscribe, you may just need to explicitly stop subscriptions.

When you are subscribing, it is very important to ensure that you always call .stop() on the subscription when you are done with it. This ensures that the documents sent by the subscription are cleared from your local Minimongo cache and the server stops doing the work required to service your subscription. If you forget to call stop, you’ll consume unnecessary resources both on the client and the server.

There is a “get out of jail free card”, though:

However, if you call Meteor.subscribe() conditionally inside a reactive context (such as an autorun, or getMeteorData in React) or via this.subscribe() in a Blaze component, then Meteor’s reactive system will automatically call this.stop() for you at the appropriate time.

It may just be worth checking how you make use of Meteor.subscribe().

5

We are using Meteor.subscribe inside of getMeteorData with React.

Subscribes have been slower than calls in any case, so we have been slowly migrating over to calls.

I had one interesting case where google was indexing the site and hit thousands of pages quickly and the server ran out of memory. In that case, as a quick fix, I changed some of the publish function on the server to use

collection.forEach(doc => this.added(‘collection’, doc._id, doc));
return this.ready();

That seems to help out.

6

Any query returning all documents is likely to end up with poor performance and resource issues. If possible, you should only publish the documents (and fields within) that the client needs at that time. The difference between what you have there, and what I assume you had before (return collection.find()) is that your code just returns a snapshot of the collection.

As you’ve discovered, that’s really no different than a method call (except it’s less efficient).

7

Sorry, I was just giving you a highlight of the code. We are very careful to only return the records that are needed by the client. We pass criteria down from the client and limit result set sizes from mongo and use this.userId where we can on the server for security and reducing result sets.

collection.forEach(doc => this.added(‘collection’, doc._id, doc));
I assume this is less efficient because I fetch the collection and then have to loop through that set again? But no observer will be created?

8

No - it’s because pub/sub returns each document (or change to each document) separately. A method can return a whole collection (if you’ve got the memory, bandwidth and patience).

9

I’m glad I posted this question. Makes a lot of sense. The data has to get pushed into minimogo.

Thank you for your help.

1 Like