Hey,
after upgrading from Meteor 1.6 to Meteor 1.7 I’ve noticed a big memory leak in production (process eats ram until it crashes). When we revert back to 1.6, the problem disappears. Anyone else noticed that issue? We are running Meteor on a Ubuntu 14.04 machine with latest Node 8.12.
I believe I was having the same issue. After upgrading from 1.6.1.1 (which is working fine for me) I started noticing Memory leaks in 1.7.0.3. I took many heap snapshots (which I can upload if it would be helpful) using the heapdump NPM package. While I’m not deeply familiar with using a memory profiler on NodeJS applications, a couple of things jumped out. First, an ever growing Error object (frequently 200MB+ at the time of crash) and an ever growing pool of Closures, many linked to the Fibers library, normally over 900,000 closures. After failing to pinpoint the exact issue and rolling back the 1.6.1.1, the issue has been gone for 4 days now. I should also point out that it was hard to get this issue to trigger in development, and I only ever saw the memory leak manifest in production. This could just be because of the amount of time needed for the memory leak to manifest (usually around 6-12 hours in our experience) or due to some set of conditions triggering the leak in production but not in development. All I know for sure is that rolling back to Meteor 1.6.1.1 fixed the issue.
Hi. We are experiencing the same issue in our production setup.
In Meteor 1.6.x we have no memory leaks, but upgrading to Meteor 1.7 (and 1.8) we have a massive memory leak of fibers (and closures related to those fibers).
Doing heapdumps shows us millions of fibers being retained (after running with load for ~1 day), but with no distance to the root GC node (distance is shown as a dash in Chrome devtools).
In Meteor <= 1.6, memory dumps only contain up to 20 fibers no matter how long and how heavy load I have tested with.
We have managed to reproduce the leak in development by simulating a few (20) concurrent users calling Meteor methods and making subscriptions on both Meteor 1.7.x and Meteor 1.8.
I suspect the root cause is related to this old fibers issue in which the node process is permanently hosed with fibers whenever a larger number of fibers than the size of the fiber pool (default is 120) are active at the same time. At least, the speed of the leak was reduced considerably when I did some steps to run fewer fibers in parallel (our node process is reading from a rabbitmq using the amqplib npm package and I reduced the prefetch count so fewer messages were being processed in parallel).
It might also be related to the fiber pool that the meteor-promise npm package. That pool has a size of 20, which is hardcoded when Meteor starts up. Whenever more than 20 fibers are running in parallel, the pool will be empty and the package creates a new fiber - however, when a fiber finishes, if 20 other fibers are already idle, the pool’s target size is already met and that fiber is just “released” so it won’t be reused. And when I look into the retainers in my heap dumps, I can see the millions of fibers there are retained via the fiber variable in line 14 in fiber_pool.js of that package. So maybe those closures (of the makeNewFiber function in that file) or something else is keeping a hold of all those fiber objects that won’t be reused whenever the condition I just described occurs.
As discussed in:
Please go to Github where this is being addressed like here:
That issue has nothing in common with our issue.
That issue is related to Node 8.12, which neither Meteor 1.7.0.x nor Meteor 1.8 uses (yet). It isn’t related to a memory leak either, but instead to excessive CPU usage related to GC changes in Node 8.12.
This other issue sounds like it might be related though: https://github.com/meteor/meteor/issues/10032
Is anyone aware of a github issue about this (seemingly fibers / meteor-promise related) memory leak?
I haven’t found anywhere else where anyone is discussing or working on this issue and we are currently blocked from updating past Meteor 1.6 because of it.
Probably go ahead and create an issue if you don’t see any there that mentions this.
I had similar issue which was reloading my client constantly after few minutes. Fixed it by consolidating my subscriptions, but I haven’t fully investigated so I don’t know if it is related to your problems.
Found this, maybe related:
I have created a new issue on Meteor’s GitHub: https://github.com/meteor/meteor/issues/10359.
Please upvote / add more details if are experiencing the same issue.
1 Like