High CPU Utilization Percent

[cubicsinc]

We are using different databases to store data of company. For example, user’s general information (login, password, profile info, etc) stored in the main database, but user’s messages, notifications or any other data stored at the separate database for each company.
Connection to the database with publication of messages looks like this:

Meteor.publish('messages', function(companyId, query, limit){
	let mongoDbUrl = 'mongodb://IP_ADDRESS:27017/' + companyId;
	let CONN = {};
	let getDbConn = (dbUrl)=>{
		let database = null;
		if (CONN[dbUrl]) {
			database = CONN[dbUrl];
		} else {
			database = new MongoInternals.RemoteCollectionDriver(dbUrl);
			CONN[dbUrl] = database;
		}
		return database;
	};
	let dbUrl = mongoDbUrl + companyId;
	let database = getDbConn(dbUrl);
	let messages = database.open('messages');
	return messages.find(query, {limit: limit});
});

Furthermore, we are using rocketchat:streamer package to inform user that another user is typing message just now.

After 100-150 user connections on production server CPU is running above 90%. How to reduce CPU utilization?

[msavin]

You’re initializing a new Mongo connection inside the subscription, meaning every time that publication starts, a new connection to the database is made. Instead, initialize the collection outside the publication.

There are also a few security issues with this publication, but I’m assuming you are aware of them.

[cubicsinc]

No, I think it doesn’t create new connection, because the function getDbConn checks whether there is connection to this database or not.

[robfallows]

No, @msavin is correct. Each client that subscribes instantiates a unique “copy” of the publication, so that’s one connection per user.

[cubicsinc]

So, how to correctly rewrite logic of the connection to the separate database?

[robfallows]

That’s harder and it’s one reason why multi-tenancy within a single database is used so much.

Assuming the work needed to adopt a single, multi-tenant database is too much, you could try caching the connections on a per-company basis - an array of connections indexed by company, if you like. You won’t achieve optimal re-use in a multi-server environment, but it should be way better than you’re getting right now.

You will need to destroy connections when all subscribing clients to that company have disconnected, so it’s not an ultra-simple solution.

[znewsham]

Another solution might be sharding - you access the database from a single URL, but you can store the data in distinct replica sets - one per shard. Note that this is also generally discouraged, as it doesn’t scale particularly well (e.g., when you have one massive company, and 1000 smaller ones, they all get one shard).

A lot of this depends on what you are hoping to achieve by having one DB per tenant?

[jamgold]

Correct me if I’m wrong, but doesn’t storing/caching the connections in a variable CONN that is scoped inside the publication function mean every new publication to the same company will create a new DB connection, no matter what, since CONN will always be empty?

[ramez]

@cubicsinc, I agree with @robfallows, why not have a field company in collection messages that is indexed. I can’t see how this approach is scalable. This is how we keep data between multiple clients separate.

Secondly, mergebox will kill you. You have a lot of data being sent down. If this is the only time this collection is called, consider using disable mergebox package. Also, why do you use streamer, why not inside your pub call this._session.sendChanged or this._session.sendAdded (see more here: https://github.com/meteor/meteor/issues/5645)

Finally, if you MUST have multiple dbs, store the handlers externally like @jamgold mentionned. `

[marklynch]

That’s what I thought too. Maybe @cubicsinc you should make CONN global and then you should only have one DB connection per tenant. Not sure if I missed something else but that seems like an obvious error to fix first. You’ll probably still get high cpu for lots of users anyway depending on what container you’re using.