Meteor v2.8 - Memory leak in mongo driver?

core
21

No, I don’t have any. But I saw an extensive discussion here and on MongoDB Jira already, so I guess the problem is (most likely) already identified.

Thanks for that! I quickly checked the mongo package sources and I think it won’t be hard to delay cursor creation up until it’s needed. We could try that and see if it’s still a problem.

However, I’m super curious if something as simple as .find() without consuming it is also a problem with the driver itself. If it does, and it wasn’t the case in the older versions, IMO it’s something they should fix (or, at least, document properly).

22

Find without consuming certainly triggers the issue here (in 2.8.1) but I’ve confirmed that on 2.5.7-beta.0 (our production version) it does NOT. So this does look like an API change in mongo - I doubt they’ll be interested in fixing it though since it’s pretty unusual to create a cursor that you don’t then use - but yes, better documentation of this change would be nice

23

created this PR: Make count NOT create a cursor by znewsham · Pull Request #12326 · meteor/meteor · GitHub

4 Likes
24

Thanks a lot everyone for all your quick support and even the PR just created hours later - great stuff! I love this community! :heart:
Fingers crossed this also fixes our RAM problems - we are currently already deploying a hotfix ourselves:

Mongo.Collection.prototype.countDocuments = function (query, options) {
  const coll = this.rawCollection();
  return Meteor.wrapAsync(coll.countDocuments.bind(coll))(query, options);
};

We will give you an update if we still run into any issues.

1 Like
25

Supposedly fixed in 2.8.2 from the PR of @znewsham :tada: Make count NOT create a cursor by znewsham · Pull Request #12326 · meteor/meteor · GitHub

3 Likes
26

Update from our side:
After the hotfix deployed yesterday, our RAM problem indeed got a lot better! :slight_smile:
BUT: It seems we are still leaking (-> not closing) Mongo Sessions somewhere!

We implemented a method to log activeSessions like so:

const client = MongoInternals.defaultRemoteCollectionDriver()?.mongo?.client;
const sessionsCount = client?.s?.activeSessions?.size;

This is number is still climbing ALL the time and NOT going down!
It has been 5 hours since the last restart of this instance and with peaks of only ~100 concurrent users, we are already at 14k (!!) active sessions - and still climbing (although slowly).
On the 2nd instance (also serving the same users - we have a traefik loadbalancer in front of it), we are already at 17k. So it basically means that this load over the last 5 hours managed to open and not close a total of 31k Mongo Sessions!

Can anyone still confirm this? It seems to me there might be other bugs/changes in the new mongo driver we missed?

We are, obviously, running Meteor v2.8.1 and we patched ALL .find().count() in our codebase with the small patch I posted here.

Update:

I think we found the problem! It is in the very common package meteor-roles … and OF COURSE it is .find().count() again! :grin: look here: meteor-roles/roles_common.js at master · Meteor-Community-Packages/meteor-roles · GitHub

Update 2: And indeed - we found it!!
After an update to Meteor 2.8.2 the problem is solved!
→ We can confirm that v2.8.2 solves the issue!! Good job everyone!
Now our Sessions stay at a solid 0! Heureka!! :grin:

10 Likes
27

Great job guys identifying and fixing an important issue! Many thanks to everyone involved.

1 Like
28

Hello,
If we update to v2.8.2 do we have also to do a patch like you did (or updating to v2.8.2 is enough) ?

1 Like
29

What it seems is yes. Updating to 2.8.2 can solve.
Awesome to see guys! Also, if you guys want to upgrade to the latest mongo driver version there is one rc candidate out for 2.9.

30

No need for our custom patch, updating to 2.8.2 should be sufficient!

4 Likes