Setting number of MongoDB connections

help
1

Is it possible to set the number of connections each Meteor instance makes to MongoDB. We have a little under 20 nodes connected to our MongoDB and they use more than 350 connections which means we have to pay for a higher tier of service to MongoDB Atlas which we’d like to avoid.

Is there a way to limit the number of connections each node makes to the database?

Separately, after vertically scaling up our database, the number of connections dropped from 350 to around 130. The number of connected servers to the database didn’t change, but the connections dropped massively. Could it be there were some hanging connections that got wiped when we scaled up? Or does the amount of RAM/CPU that the database has available to it determine the number of connections too? This latter question may be better suited for a MongoDB forum.

2

You can use Mongo.setConnectionOptions and pass in the poolSize parameter to set the number of connections in the pool. The default is 5. For example: Mongo.setConnectionOptions({ poolSize: 10 }).

2 Likes
3

Given how complicated it is to correctly use Mongo.setConnectionOptions (one needs to create an internal package to correctly schedule the change of configuration before any use of the configuration), a simpler alternative seems to be to use Meteor.settings: METEOR_SETTINGS='{"packages": {"mongo": {"options": {"poolSize": 10}}}}'

PS: I mean, it’s so hard that someone made a Meteor package to automate setting it up.

1 Like
4

I guess you can just add it to your MongoDB URI. Example of options:

const options = '?retryWrites=true&maxIdleTimeMS=5000&maxPoolSize=30&readConcernLevel=majority&readPreference=secondaryPreferred&w=majority&heartbeatFrequencyMS=15000'
5

I have recently submitted a PR to allow for setting of minimum pool size for oplog:

That is a good speed optimization. As for maximum pool size it is the same principle. Probably the issue can be growing number of oplog connections.

You will have to closely monitor and tweak the values to see how your app behaves to receive the desired result. You can set them via Meteor settings:

{
  "packages": {
    "mongo": {
      "minPoolSize": 10
      "maxPoolSize": 40
    }
  }
}

Code (and comments) around where the options are applied:

// Internally the oplog connections specify their own maxPoolSize
  // which we don't want to overwrite with any user defined value
  if (_.has(options, 'maxPoolSize')) {
    // If we just set this for "server", replSet will override it. If we just
    // set it for replSet, it will be ignored if we're not using a replSet.
    mongoOptions.maxPoolSize = options.maxPoolSize;
  }
  if (_.has(options, 'minPoolSize')) {
    mongoOptions.minPoolSize = options.minPoolSize;
  }

See the following article for reference:

I have looked into this question as part of my help desk series, if you like what I’m doing please sponsor me.

6

I’ve looked through the code base (Meteor v3.0.1 mongo package) and can’t see poolSize being set anywhere, so at best that option has been discontinued.

7

Note that those comments specifically hint at the fact that the purpose of this code is to override user settings for the oplog connection. If you dig a bit deeper, you realize that the oplog tailing only uses a single connection.

1 Like
8

Are you sure? Aren’t most of those options eventually passed down abstractly to the Mongo driver without explicit reference?

Of course, it’s probably preferrable to use minPoolSize/maxPoolSize anyway.

9

Has someone tried this?

10

Yes, that is a pretty good possibility, should have thought of that. The question then arises how many of those get applied and which get overriden by Meteor. Will have to dig deeper into this.

I do add some stuff in my MongoDB URI, it was hit and miss with them, best to add as many options via settings, I think in many cases those will take precedence, which is probably a good idea via options/settings it can react to specific setup of the app rather than hard coded values in the URI.