This code makes mocha test run out of memory on meteor 3.1.1

help
1

Hi all,

since I updated to 3.1.1 my mocha tests run out of memory. I isolated a function that causes this, and I am really curious why this is only happening on 3.1.1 (on 3.1 and prior to that version, everything’s fine).

The functions restores demo data to the MongoDB in order to run tests on them and is called in the beforeEach hook:

const collections = [... an array of 20 Meteor MongoDB collections...]

export const importDemoData = async () => {
  for (const collection of collections) {
    const dataString = await fs.readFile(
      process.env.PWD +
        "/demodata/" +
        collection.rawCollection().collectionName +
        ".json",
      "utf-8"
    );
    const data = EJSON.parse(dataString);
    await collection.rawCollection().insertMany(data);
  }
};

The Ejson files are ~65 mb in total. Tried to trigger gc manually already, did not make a difference.

Besides from the question why this fails on 3.1.1, I’m pretty sure that there’s a more elegant way to get a demo db restored from some files…

2

How about the normal route of restoring backups using existing mongodb tools?

1 Like
3

So inside a mocha test, you mean using execSync and mongorestore for example? This would definitely solve the issue, and this should be much more efficient. Will try to implement that.

However, do you have an idea what could have changed maybe in node or mongo driver regarding memory consumption? Would be nice to know what design patterns are causing trouble now…

4

I’m not knowledgeable about the changes. But it definitely looks like a memory leak. You can try explicitly unassigning the variables or setting them to null.

5

Already did that, also tried to force gc.

In the meantime I implemented your approach, which works fine for restoring the db. But it seems like I came up with a wrong conclusion when I first thought that the above code would cause the memory leak.

Because now, although I do not use the above code anymore, the memoryleak is back in the tests again.

I guess that as I removed restoring of the demo data, certain tests that cause the memory issue just failed. Will have to investigate further…

UPDATE

I tried to profile memory usage. My beforeEach hook looks roughly like this:

beforeEach(async function () {
      logMemoryUsage("Start before each");

      importDemoData(); // sync exec mongorestore, prev. via bulkInsert into collections. 


      await new Promise((res) => setTimeout(res, timeout));

      const currentUser = await Meteor.users.direct.findOneAsync({
        username: "testadmin",
      });

      await Roles.createRoleAsync("admin", { unlessExists: true });
      await Roles.addUsersToRolesAsync(currentUser?._id, "admin");

      sinon.stub(Meteor, "user");
      sinon.stub(Meteor, "userAsync");

      Meteor.user.returns(currentUser);
      Meteor.userAsync.returns(new Promise((res, rej) => res(currentUser))); 
      logMemoryUsage("End beforeEach hook");
    });

If I set a timeout to eg 8000 milliseconds, the tests do not run out of memory (at the end of the test suit I’m at constant 828mb heap used); without waiting for the timeout I end at 1280mb heap used.

Both implementations restoring the demo data have in common that they are quite busy on the database of course, and both leak memory if I do not wait for the timeout. Once I set a timeout I have no memory issues anymore.