Subscription caching - how to suspend subscription

Hello,
I am playing a bit with cached subscriptions and I wonder if there is some way or a hack for subscription to be suspended. What I want to achieve is when going to another page I suspend subscription which won’t receive new updates, but still keeps loaded data. Once I go back to the page that needs that subscription data is immediately available and I resume listening for new data.

I’m not sure this is possible natively in meteor. Though it may be possible to fake something. The problem is that while the subscription is paused (assuming that were possible) the server would need to continue observing and caching the events so that when the subscription is resumed, the server can send the delta to the client. As such, the only thing being saved is transmission cost, and even then the server will eventually have to send some days to the client (potentially the same amount of data as it would have originally) so the server has to work almost as hard so there is limited value.

If you want the subscription to persist outside of a single template or page, you can look into using a subscription manager.

Can you describe your scenario? It could be that there is a better way

In early Meteor days, there was meteorhacks:subs-manager that allowed caching like this. I just did some research and found this more recent alternative:

Haven’t tried it myself yet, but maybe it does the trick.

1 Like

That’s what I am using and modifying to my needs. However cached subscriptions are all “alive” and would receive all changes behind the scenes so I would like to change that.

Yeah I know there is no API for such thing so I kinda thought maybe there is some hack I am not aware about (long shot I know). The idea would be that server would do all the calculations for delta only when subscription is resumed - it makes no sense to do it real time for unused subscription. Otherwise caching is not that beneficial as it would put a huge load on the servers (if you cache a lot) when in reality very likely that cached subscription isn’t even used anymore.

BTW scenario is simple and pretty common - you have page A and page B both subscribe to a lot of documents. When you leave page A subscription is stopped in normal flow so when you go back to it from page B you need to resubscribe. You can cache subs, but then all those subscriptions in the cache are still alive and put huge load on the server making this client side performance improvement most likely not worth it. So this is more of fundamental question (or I would say now feature request) than some scenario specific issue. I was kinda surprised that there are no way to “suspend” subscription - seems such a basic feature. I guess it is kinda implemented for connection reconnect, so maybe it would be possible somehow to redo this without losing connection… But I guess server side subscription is still active real time in that case like in your scenario so it’s not exactly what I have in mind.

You could create a local collection and manage it by a real collection observer. I imagine something like this.

import RealCollection from '/imports/api/RealCollection';

// create client side minimongo collection without connection to the server
const CachedCollection = new Mongo.Collection('cachedCollection', { connection: null });

// on component mounted
const sub = Meteor.subscribe('real-collection-data', { onReady: () => {
  // check which documents were removed in a mean time and remove tham manually from the cache
  CachedCollection.find({}).fetch().forEach((doc) => {
    if (!RealCollection.findOne(doc._id)) {
      CachedCollection.remove(doc._id);
    }
  })
} });

// observe changes of the real collection
const observer = RealCollection.find({}).observe({
  added: upsertToCache,
  changed: upsertToCache,
  removed: removeFromCache,
});

// on component unmount
observer.stop();
// important here to stop the observer first before stopping a sub, to avoid removing docs from cache
sub.stop();

Interesting idea, but still some state is needed server side. Let’s say I cache data on local collection then leave the page, then go back and have that data in local collection, but the moment I subscribe again to real-collection-data, server will send everything again.

I think your desired use case is simply not covered in Meteor’s pubsub. You can’t suspend and resume subscriptions. You can emulate that behavior, but the amount of data transferred will not be optimal.

1 Like

Yeah I know it’s not covered. In a way closest thing is connect/reconnect but this is per whole connection, not per single subscription. I wonder how it is implemented - is server dumb and still calculates real time changes on subscriptions or it does nothing and only calculates delta when connection is restored. If it’s second option then maybe it would be possible to reuse that somehow and add suspend for single subscription which could open up interesting uses for performance gains.

I think the subscription will be terminated as soon as the connection goes down. Upon reconnect the subscription reruns, but then it’s a new subscription.

Could be, it’s not exactly clear from documentation. In that case it’s a simple primitive implementation that is not really useful for my case (and probably not as useful for disconnect/reconnect cases as it could be). Anyway maybe it could be a feature request for the future…

What you want to achieve is not really possible. You want a server to stop tracking and reporting the diff, but then you want to start tracking without syncing data again. Someone has to do the book keeping, unless you know that data can only be added and you don’t expect any changes/removes of data.

If that’s the case, you could simply use meteor method instead of pub/sub and implement some kind of pagination.

But if data might change or be removed, you need a pub/sub. If data is not user specific (multiple users subscribe to the same data) it might be more efficient to keep the subscriptions open. Meteor reuses cursors, so if you would have 1 user subscribing to a dataset, then when the next user subscribes, meteor will send the data from the cache instead of querying database again. In that sense it might become more efficient to keep the subs open. Depends of course on how often user switches from page A to B and back again.

I subscribe, server returns elements A B C D. I suspend this subscription - server saves current state and does nothing (until resume or timeout). Client resumes subscription, server reruns it and sees that E was added and B was removed so it sends only add E and remove B ddp messages. Of course if meteor reuses cursors in certain cases it’s not worth it so it would be on case by case basis.

You’re right, this could work – if someone adds this functionality to Meteor, or, alternatively, sets up a parallel pub/sub.

^^^^ A caveat here could be that if there are plenty of clients with lots of documents (and/or large ones) to be suspended, this could cost lots of RAM on the server, if the suspended subscriptions’ state is to be saved transiently. Alternatively the suspended state could be saved persistently, say, in redis, where each entry can have a TTL. But you see now it starts to get complicated.

The problem with such approach is that in order to compare changes after “resume”, you would still need to fetch entire data set from the database to the server. Then run a loop to deep compare all cached documents with newly fetched ones. If collection is big, it will block your event loop. So doesn’t sound like an optimisation to me.

If you’re talking about large collection it will be better to keep the subscription open and let meteor only track deltas.

I am not sure how exactly meteor is implemented behind the scenes, but doesn’t server track state already with merge box? Question is can we save CPU cycles and bandwidth with “suspended” subscription. Of course if you start overusing and cache too much then memory would be a big problem.

Well blocking event loop is of course not an option. Maybe it could be done in worker thread, remember we already have old data on client so it’s not a big deal if updates arrive after slight delay. Deep compare is done anyway for running subscriptions for each event, in this case it would be done on potentially many events or if suspended subscription times out it won’t be needed at all.
Anyway fundamentally suspend/resume events/messages/streams is not something unimaginable and impossible to do even if there are problems to solve (which why it is interesting).

You are correct, meteor’s mergebox maintains the state of each clients subscriptions on the server, specifically the exact top level fields and their values for each document. There is some debate over whether this is a good thing, it causes high memory usage for somewhat limited value in many situations and causes trouble when you want to subscribe to a changing subset of non top level fields, but regardless, that is how it currently works.

How this relates to pausing subscriptions is complicated. The first thing to note is that subscriptions aren’t quite an event stream. Meteor sends deltas to the client, not events, and not entire documents, so to implement pausing you’d need to do one of a few things

  1. Cache not only the current client state (as it already is), but also the state the client should be in and then send the delta when the subscription resumes - this will cause an increase in memory, no decrease in CPU and a potential decrease in network transmission (and the associated CPU costs)

  2. Cache the current client state (as is) and also cache the observed events and replay them. This is likely worse than (1) in memory but cheaper in CPU.

  3. Cache the current client state (as is) and re-run queries on resume to determine the correct state - in this case CPU cost is 0 during pauses, but potentially high (and high db costs. Which are harder to scale) at the time of resumption.

There are probably other options. I’d be interested in hearing them and their associated CPU/db/memory costs.

The problem is they’re all of limited value in most scenarios. Unless you’re collections are not only hot but “toggly” there’s limited value in pausing a subscription vs just continuing to observe the events. The only situations I can see that would benefit from this scenario are:

  1. You’re looking at an actual event stream - the documents don’t change but new ones are constantly added

  2. A value is set then unset (or similar) and you want to debounce

For (1) I could see some value in implementing this in the core. But it can easily be accomplished using methods or even a combination of methods and server side observers if you really wanted the shared cache of publications. For (2) this seems highly specific but could probably be accomplished with a custom publication handler with debouncing (I’ve done similar when I want to update counts reactively which can be expensive)

Does the above address your scenario?

1 Like

Thanks for that overview @znewsham. Seems there are mostly no gains server-side unless for very specific cases.

Given this, there can still be value of caching on the client side.

Visit page A → pub/sub to tons of docs → move to another page but keep cache in minimongo → go back to A → immediately load what’s cached from minimongo → pub/sub to same publication again → replace records in minimongo as they arrive and reactively update UI in page A as needed

Caching in client; no change in server-side publication