Thousands of simultaneous subscription to the same publication?

Performance-wise is it ok to have thousands of simultaneous connections to subscribe to just one publication?

We have a mobile app with app settings that are being managed in an admin panel. All app clients are caching the settings. To ensure that changes to the settings are propagated to all mobile app clients, we created a publication serving the hash of the settings. Whenever the settings are updated, the hash will also change, and this will trigger the clients to fetch the new settings. In the future, it is possible that thousands of clients will be subscribing to this publication in real time. Is there any performance issue that we might face in the future with this setup wherein once change can trigger subscription updates to thousands of clients at the same time?

I think a better way to think about this is to consider if too many users are connected to publications on a single server in general, and not if too many users are connected to the same publication. Performance degrades quickly when the server is watching for updates for too many clients. I could be wrong, but I don’t think there’s much of a difference between 1,000 clients -> 1 pub, vs 500 clients -> 1 pub and 500 to another pub; it’s still 1,000 clients for the server to check if an update is relevant specifically to each user subscribed.

Of course, the activity within a specific publication has some effect, which is what I think is the root of your question (is it constantly changing?), but in general, it’s best to ask yourself if this needs to be specifically reactive. Do all clients (that happened to be connected right at the time of the update) need to see those changes immediately?

My suggestion is as follows:

  • Create a method to fetch the settings from the db. Call this method when a template loads and pass it to a helper, some reactive var/dict, or just cache it regularly - wherever it needs to go.
  • Set a reasonable interval to refetch for new settings (every 15s? 30s? 2m?). Meteor.setInterval(() => Meteor.call('method') , 60 * 1000);

If all users are supposed to get the entire update of the col (i.e. there’s no specificity around certain docs in a col for specific users, or even within a certain doc, etc.) then you’re really having the publication work on overload for no reason. On an update to the handle the publication is instructed to observe, it checks to see for which of the connected users the update is relevant and then publishes the update to each of those users. It sounds like in this case, the whole update is always relevant to every user, in which case you should just use a method on an interval so the mergebox logic isn’t running its hat off for nothing.

Also, consider if most clients are always connected or if they frequently exit/re-enter. If they’re not always connected, then you can have this method run on the router level or some high-level general template (or layout) on load. Maybe if they’re on and off frequently, you only ever need to do that once each time they return. If they’re usually connected for longer durations, then the interval -> method implementation is a good solution.

In short, you are probably better off not forcing the server to watch for changes for 1,000 individual clients, and instead having each active client call a method on a 1m interval.


Separately, I would take a look at redis-oplog. It’s worked wonders for us.

1 Like

Good to know that redis-oplog is working wonders for you. We are also using it. It allowed us to easily trigger reactivity from one app (admin panel) to a different app (mobile). Fantastic package.

Thanks for the approach you posted. I’ll keep that in mind.

As of now, we are a little tied to the required reactivity because HCP, although it has improved in the recent beta, still has quirks resulting for the mobile app to be stale requiring updates from the respective app stores. So when we release changes (e.g. changes in methods or collections) resulting to incompatibilities from previous app versions, we need to make the incompatible apps show a notice gracefully instead of the users getting different errors.

Good point about the number of servers. We are hosted in AWS and has autoscaling capabilities. There is just no way for us to test thousands of real time connections and how does it look like with a meteor publication.

How about a simple function you can put at the top of each method that checks for version compatibility and throws the notice you mentioned if there is a mismatch?

Write the currently installed version as a static client-side variable. Then you have two options as I see it:

  • Pass it each time you invoke a method and then pass it to the check function a the top of each method. Or,
  • Write the installed version to the user’s profile so the method can easily access it with Meteor.user(). This way, you only need to add the version check function to the methods themselves, and don’t also need to change all your existing method invocations. (If you do this, check out userCache by msavin to severely reduce the # of db calls on Meteor.user().)

A plus to this solution is that it can even work on the very next server upgrade, even though the version variable is not yet on non-upgraded clients, because if the version var is missing from the method call (either via a param or Meteor.user(), depending on which you choose), you can treat that as a sign of version incompatibility!

Either you can make it system-wide (i.e. if there is a version mismatch, every invocation fails - good if you want to keep it simple and just make your users always upgrade), or you can create a clean static var serverside (or in the DB I guess… but that means a guaranteed 1 extra call to DB for each method) that has version requirements recorded for each method.

Ex.

const versionCompatibility = {
   'methodOne' : 1.2,
   'methodTwo' : 1.6,
   ...,
}

function checkForCompatibility(methodName, currentVersion = null) {
  if (!currentVersion || currentVersion < versionCompatibility[methodName]) throw new Meteor.Error('Graceful error msg here', ...);
}

Meteor.methods({
  'methodOne'() {
    const { currentVersion = null } = Meteor.user('profile.currentVersion').profile; //Uses msavins lib
    checkForCompatibility('methodOne', currentVersion);

    ...;
  }
})
1 Like

As mentioned in the article https://guide.meteor.com/data-loading.html#lifecycle

  1. The server sets up a query observer on that cursor, unless such an observer already exists on the server (for any user), in which case that observer is re-used.

I think you shouldn’t have any problem in the future, if you have publication with static query to mongodb and your collection contains one doc with all configs. Observer will be re-used and only one doc will be pushed to all clients.

You can switch from pub to method at any time. Just create facade for your configs (e.g. ConfigService with get reactive method) and incapsulate logic about retrieving information.

1 Like

Thanks. Will look at the userCache package

Great to know this. The actual fetching of the entire config set is now setup as a method. So the publication is just one document that contains the hash of the entire config set.

We are now expanding this implementation. Currently, we also experienced the continuous refresh of web apps when they were left in the background and then we released a new version. Through the same mechanism, we are now forcing all apps to disconnect when deploying and then reconnect after 45 seconds just to ensure that our server instances have already successfully switched to the new deployments

1 Like