Memory usage on prod ~5x that of dev

[sbeaton]

Hey all,

I’m working on a meteor app I inherited from the previous dev, so I’m not quite an expert on meteor.

I’ve got the app running on heroku, with hobby dynos. There’s 2 set up, one for staging and another for production. The staging one has a similar database to the one in production - I copied it over not too long ago to do some testing with real data. The memory usage of the staging app is around 80-130mb. It can go up a bit (~20mb) when you’re messing around on the app.

The problem is with the production version. It sits around ~650mb, minimum 550mb, with spikes over 1gb (when according to heroku, there were 5 requests/min).

I can recreate this issue on my local dev env as well. When running the app with meteor, it uses just under 100mb (as told by ps aux). When running meteor run --production, it hits over 500mb, just like on heroku.

There appears to be something that’s causing the app to use a ton of memory whenever it’s in “production mode”. Can anyone tell me where I should start looking to track down this issue, or if you’ve had the same issue, how you’ve fixed it?

Thanks in advance!

[wildhart]

To start with I would search all files for Meteor.isProduction and Meteor.isDevelopment to see if there are any conditional code differences between the two builds.

Also try running with the --inspect --production flags, connect to the node debugger using Chrome Dev Tools (chrome://inspect/) and take a heap dump. Often this causes node to crash though, but you might get lucky. If it works, inspect the heap to see where the memory is being used.

[storyteller]

What version of Meteor are you using?

[sbeaton]

I’m using Meteor 1.8.1

[sbeaton]

@wildhart I’ll take a look at the heap dump option. There’re no instances of either Meteor.isProduction or Meteor.isDevelopment in the code, though.

@storyteller not sure if the I replied correctly on the previous post. I’m using version 1.8.1

[tosi]

Installing and configuring Kadira gives nice insight into the app (a bit scary when you look at the instructions, but there is a docker image that my colleague pretty much used as is and I just connected the frontend which took about an hour of trial and error).

Isn’t it related to the users? On your local dev it is probably just you, but every user on prod is consuming resources, especially for publications. My colleague recently created very funky function can got called many times and created many subscription handlers that were never destroyed…

[sbeaton]

@tosi I looked at that, but it seemed like it shut down not too long ago. Are you talking about the self-hosted version?

I’ve done some more debugging, and I created an additional app on heroku to see if there was some issue with the original app - I was thinking the original app might be cursed or something!

I created it, set it to use the same database as the other production app, and didn’t have the performance issue. However, as soon as I moved the custom domains over from the old app to the new one, I started getting the performance issues. Also, once the domain was removed from the original app, its resource usage dropped to half of what it was. I reverted the changes back, and the original app went back to it’s high resource usage, and the new app dropped back down to ~120mb.

So it seems like something is happening with heroku’s custom domain feature, and how it’s interacting with meteor. I’ve opened a support ticket with them, but if anyone here has a clue as to what’s going on, please let me know!

[tosi]

Kadira: yes, you will have to host it yourself. Or pay for it by e.g. hosting on Galaxy.

[pmogollon]

You can use monti apm.

You might want to check if you are using oplog, if not that can be the issue.

[wildhart]

I can also highly recommend https://montiapm.com/. Even the free version is really valuable, and the paid version well worth the small extra cost. Super easy to set up.

Back to the problem though, @sbeaton, didn’t you say that you got the same problem on your localhost compiling with the --production flag? That should eliminate any resource usage related to user connections or heroku’s custom domain features.

If you can you easily reproduce the issue on your localhost, even without any clients connected, then try removing all your server-side code ( /server, /lib and any imported /import files) and see if the problem goes away, then start incrementally adding it back and see when the problem comes back again.

If the problem doesn’t go away even with zero server side code then start looking at your packages - remove as many meteor and npm packages as you can and see if it goes away, then add them back.

If you still have the problem, even with no server code and minimal packages then try wiping the database ("meteor reset") and see if the problem goes away. If it does go away then restore the database (make a backup of /.meteor/local/db/) and selectively drop each collection until the problem goes away again ("meteor mongo" then "db.<collection>.drop()").

[wildhart]

@sbeaton, did you get to the bottom of this?

Can you reproduce the problem on localhost with the --production flag?

[evolross]

Bump to OP - don’t leave us hanging!