Working with users

performance
1

Hello, I’ve learned that parts of the application which will have peaks of traffic should not user the publish subscribe, but fall back to the normal rest methods (that would be Meteor.call(…), and prepare on the server side the proper methods).
It would be nice for the app I’m working on to have at least 1 publish subscribe feature, where an admin “changes page” and all connected user “see the page changing”. Users can download all the pages upfront, it would just be a subscription to a number. Is this any better? Could this work with 1000 users? Is there any recommendation?

Thank you very much, T.

2

Yes, that’s a sensible approach.

If you can arrange a single document with a single piece of data, like a version number, to which the users subscribe to in order to find out when to fetch the actual changes, it is certainly better than to have them subscribe to lots of documents containing lots of data.

You can further improve performance by using a server side caching of some larger chunks of data you know up front your 1000 users are going to fetch them very soon. But be careful, synchronizing a server side cache with intermittent database changes can be very tricky if you have multiple Meteor server instances. While this can dramatically improve performance of data fetching, you could end up delivering stale data, if improperly implemented.

2 Likes
3

thanks I was just reading your comments on the question “heavy mongo processing”, I’ll look into caching , the point would be not to query mongo when the 1000 requests hit, correct? I see the issue of old data laying in the cache. Thanks again!

4

Exactly.

Here’s a naïve implementation of the caching, and at first sight it looks quite straightforward:

import cache from 'js-cache'; // https://github.com/dirkbonhomme/js-cache
const yourCache = new cache();
/**
 * Set this to a duration long enough for all clients to call this method.
 * Once expired, js-cache will automatically delete the item.
 */
const dataTTL = 15000; // 15 seconds

new ValidatedMethod({
  name: 'setData',
  validate: new SimpleSchema({
    data: Object, // make it nicer
  }).validator(),
  run({ data }) {
    this.unblock();
    const version = Date.now();
    yourCache.set(version, data, dataTTL);
    MyData.update( {...}, { $set: { data } } ); // whatever
    // The document the 1000 users will subscribe to
    MyDataVersion.upsert( { _id: "versionSignal" }, { $set: { version } } );
  }
});

new ValidatedMethod({
  name: 'fetchData',
  validate: new SimpleSchema({
    version: Number
  }).validator(),
  run({ version }) {
    this.unblock();
    let data = yourCache.get(version);
    if (data) return data;
    data = MyData.find( {...} );
    yourCache.set(version, data, dataTTL);
    return data;
  }
});

If you have multiple server instances, fetchData's cache management still isn’t going to be optimal on the instances other than on which setData was invoked.

The reason is that once the subscription delivers the version change to the 1000 users, they all will start hammering their respective Meteor instance to fetch the data. Collection#find is entering a Fiber for each and every user, so they will practically hammer MongoDB all at the same time with the same request.

One could make a case that it doesn’t matter that much, because MongoDB is smart enough to do some caching as well, so it won’t actually perform hundreds of thousands of times the very same query. That’s probably true, but then why do we do any caching on the server in the first place?

So ultimately this was a demonstration of how to NOT do the caching server-side. :slight_smile:

Redis is commonly used as a distributed cache; it’s lightning fast, and a TTL can also be set on each item. Here’s a Node.js redis client.

It also has functions to set and get, however they need to be promisified, and used accordingly – see the documentation.

2 Likes
5

Wow thanks again! That’s perfect!

1 Like