[Solved] Caching a subscription that's order by newest for X minutes

help
1

I have a subscription of data that’s ordered by the newest first via a datetime sort.

Data is streaming into the database constantly and so what happens is the page now reloads with the latest data every time it’s updated. This is awesome to watch but sucks for user experience obviously as you just want to see the page without it updating infront of your eyes.

How is it possible to temporarily cache a subscription so that is stays the same for X minutes and then refreshes itself? I have used memcache in the past for this but that seems overkill, is there a way to tell the subscription - don’t pull again for X minutes like a TTL?

Thanks in advance for any help on this, you guys rock

Solution here: [Solved] Caching a subscription that's order by newest for X minutes - #4 by truedon

2

You could use a debounce function (e.g. lodash _.debounce) in your subscription code to update minimongo, that should do the trick.

A word of caution, though:

Publish-subscribe to something that gets constantly streamed and updated will most likely lead to an overall poor performance of your app as your user base grows. Try to avoid this pattern. Maybe you should debounce the streaming update into MongoDB to an acceptable frequence, rather than allowing all updates to be pushed down to the clients and to deal with the problems there.

3

Thanks Peter, the data is being limited and just pulled from a collection. So if I am correct in hearing you, all I need is to make a cron that dumps the rows I want into another collection and manage it that way? Thanks for helping me think outside the box here was totally tunnel visioned. Very easy to fix with a cron then!

4

Solution:

Run a cron job external to meteor (in the actual crontab), that script populates a collection for use on the home page (and anywhere that needs a cached view)

https://man7.org/linux/man-pages/man5/crontab.5.html

1 Like
5

Why not use a method instead if you don’t need real-time data?

6

Yeh, I guess I can use a method but because cron is easier to setup and I already run cron, It’s a good solution

7

Seems like you built up a lot of technical debt with your solution of pub-sub + cron + extra collection just to have a result similar to a method.

I understand that it is easy now but the maintenance of this will be a headache in the future.

Well, if what is important is that it works, then good :+1:

8
9

Yeh, It’s only really simple if you know linux I guess. No packages, no nothing, just barebones. Everything already running in cron buddy, way better then using the packages and dont need to break the awesome reactive ddp of meteor (which is why I am using meteor to be honest, I want it to update realtime on the screen but not every darn milisecond I needed a throttle.)

If meteor added a throttle feature that would be super dope but cron is a nice approach.

10

Thanks for the suggestion, I’ve tried about 3 or 4 subs manager packages and wasn’t getting anywhere at all. I have it working now, no additional packages and just OS features so lightning fast and no overhead, no need for a maintainer to keep updating the package also which is ideal.

1 Like