The caching story

storyteller · February 20, 2024, 8:21am

Splitting this from

to talk about caching and how we would like it to develop in the future.

To sum up during my stream @xet7 mentioned that he has used various caching packages with a great success. Even the deprecated appcache (there seems to be an update for @harrison appcache as well).

For the users collection there is @msavin usercache, which seem to have inspired @wildhart mergebox-cache.

For subscriptions there is @ccorcos subs-cache, though I do recall removing it a while back for some issue we had in react with it.

For methods there is epotek:method-cache, which seem to use GraphQL DataLoader for heavy methods.

Finally there is ground:db to create a complete client-side DB.

All these solutions have one big thing in common. They have not been updated in a while.

From new things we have this PR:

github.com/meteor/meteor

New mongo package options to optimize Oplog tailing performance to include/exclude certain collections

meteor:devel ← orderlion:mongoOplogOptions

opened 04:15PM - 16 Feb 24 UTC

Twisterking

+68 -1

Hello all, Following several conversations on the drawbacks of Meteor's oplog… implementation and its limitations when it comes to scaling (see: [my forums post](https://forums.meteor.com/t/cpu-spikes-due-to-oplog-updates-without-subscriptions/60028), [a slightly older blog post](https://blog.meteor.com/tuning-meteor-mongo-livedata-for-scalability-13fe9deb8908) and a [2 year old github discussion](https://github.com/meteor/meteor/discussions/11842)), I investigated in the core a bit and propose an easy solution which, I hope, should work for many users who have issues with this. Please consider our situation at Orderlion: We have several collections, where huge amounts of data are being updated on a daily basis. On all of these collections we have not a single publication, for these we don't care about any reactivity. Due to Meteor's Oplog implementation, even in these cases a big batch insert/update of data results often in very big sustained CPU spikes on the meteor app server. Internal tests I did with my suggested implementation have shown, **that the CPU load due to Oplog tailing collections with big batch updates, can be reduced about 100x with just completely ignoring those collections**. During an import of about 600k documents in a test collection, I could reduce the CPU load from an average 25-30% down to approx. 0,2 - 0,3%. The usage is actually very straight forward: Consider a `settings.json` file like this: ```js { // ... all other settings, "public": { // all your public settings }, "packages": { "mongo": { "oplogExcludeCollections": ["products", "prices"] // This would exclude both collections "products" and "prices" from any oplog tailing. // OR: "oplogIncludeCollections": ["messages"] // ONLY watch the "messages" collection for oplog updates (and all "admin.$cmd" oplogs) } } } ``` Please feel free to comment but I really hope this can be merged soon. It would help us at Orderlion a lot and solve many unwanted server restarts (server is auto-restarting because it becomes unresponsive due to the sustained high CPU load). Cheers, Patrick

So to continue the discussion. I want to ask if I’m missing anything, what are people’s experiences and which packages should we focus on first to bring back/modernize for the biggest caching/performance impact.

rjdavid · February 20, 2024, 11:18am

Due to the limitations of the localstorage, a good path with PWA is the use of service-workers to properly cache at the very least the static files served by meteor i.e. js and css bundles. Then rules for static files in public folders like images and fonts.

Google’s workbox is a good foundation for this.

P.S. Workbox v7 requires Meteor 3 due to node version requirements

marklynch · February 20, 2024, 4:40pm

We recently started using subs-cache and seems to work great, so will probably use it more widly. Something like usercache / mergebox-cache would be really useful. We had a custom one for the user collection using grapher. Something in MCP or even better in the core would be great

rjdavid · February 21, 2024, 7:43am

@banjerluke posted this related to GroundDB which he is calling Ground Collection