How to handle a lot of data on a publication?

Right, now it makes sense to me. Great explanation.

Unfortunately, cannot help you with throttling publications. Have not needed to do this myself. Should such need arise, I’d likely start by looking into managing the publication manually via this.added, this.ready, this.changed and this.removed instead of just returning a cursor from it. Might be of help, perhaps. But not sure. The gist is here: https://guide.meteor.com/data-loading.html#custom-publication

Alternatively, if throttling doesn’t pan out or is too complex, why not solve the problem with…CSS! The whole issue here seems to be the 3-4 minutes that data is populated from outside sources. During this time there are a lot of changes to the data so probably whatever the user sees on the page in those minutes will be essentially branded as ‘incomplete’ or ‘subject to change’ anyway. So perhaps instead of displaying incomplete/jumping data to the user you could just display a loading icon and not fire up a subscription immediately (or subscribe to only a small subset of the data to keep the user’s eyes busy while loading most data in the background). Once the loading is done, trigger the subscription. I’d assume that if there’s no active subscription (or only a minimal one) during the loading process then the GC issues should also disappear, regardless of the 3rd party sources hammering the DB.

In UX land 3-4 minutes is an eternity, granted. But I’d say good visuals go a long way. For example, while the loading is in progress and loading icon spinning, perhaps fire off a method every 10 seconds to the server to query about the number of new cards received during the loading. Showing a dynamically changing text a la ‘240 cards received’ under the loading icon should help relieve the waiting anxiety. Of course, assuming this is a business app and the user will then inspect the loaded cards for a good hour or two after those couple of minutes. In social media land there is no way you’d survive a 3-4 minute loading time :slight_smile: