Meteor up with 97% CPU usage on production. I'm in panic mode

[xunga]

Hi everybody,

I’m in panic mode, my application all suddenly became very slow and the process of meteorup is running very close to 100%…

I’ve already tried to restart the application through mup restart but still the CPU levels are very high.

What do you suggest me to do? Any temporary fix that could enable me to make the application start to work and give me time to debug the issue?

I’m running under ubuntu machine on AWS if this can help…

Thanks a lot!

Edit: some more infos

[lucfranken]

Start a new ec2 server so you can see whether the issue is app or configuration based.
Install Kadira.
If you have something like Google analytics check your statistics. Do you have many users? Are you aware of that?

[xunga]

Thanks for the reply, according to Google analytics seems like there are no users connected, but still the cpu is 99%.

I don’t know if I could just stop the meteor up process on the server and then restart it again, but I’m afraid to mess up something since is production enviroment

[namirsab]

If there are no users connected, just restart the server and see what happens.

[xunga]

@namirsab

If there are no users connected, just restart the server and see what happens.

That scares me. I didn’t set up the server and I’m afraid that if I do so, at the restart something might not work again.
Is there something that you suggest me to do before restarting the server?

I’m currently running Meteor, Elasticsearch, Mongo, Kibana and Nginx. Should this services start automatically at the reboot?

[namirsab]

@xunga

I didn’t set up the server and I’m afraid that if I do so, at the restart something might not work again.

Then, try to find your sysadmin…[quote=“xunga, post:5, topic:29230”]
I’m currently running Meteor, Elasticsearch, Mongo and Kibana. Should this services start automatically at the reboot?
[/quote]
It depends on your set up, so it’s difficult to say.

[moretti]

Are you using oplog? Already had this problem (but with 0.9 at the time).

[xunga]

Are you using oplog?

I’ve tried to disable it, but as soon as meteorup process starts the cpu jumps back to 100% so I guess that’s not the problem… Thanks by the way

[rbrugnollo]

I was having the same kind of problem and solved it by revising my observables. Are you using any?

[mds4m]

Kadira has a nice CPU profiler which you can use to help debug things like this
https://kadira.io/academy/meteor-performance-101/content/meteor-cpu-profiling

[vlasky]

Try deleting Meteor and reinstalling it. We’ve seen this issue happen occasionally and this has solved it.

[xunga]

Everything went back to normal
Well not really sure what was wrong, I’m investigating deeper on what could have been wrong.

[macrozone]

Some things to check:

• check your publications, do you use a lot of publish-composite or similar? Do they often invalidate?
• do you publish huge amounts of data that are not indexed? --> add good indexes to your collections
• do you have hooks on your collection or use collection.observe on the server that invoke “expensive” computations? (aggregation or so)
• what is the state of the memory? is it full? --> maybe a leak. Full memory can also lead to high-cpu usage
• maybe also list the packages that you are using

[cskaandorp]

Don’t know if this will help you, but I had something similar at Digital Ocean. My website crashed every 2.5 days with a CPU going over 100%. No errors, no apparent reason. In the end I tried a more expensive package, with more RAM. So far, so good, the problem did not occur for more than 4 days now.

[robfallows]

@cskaandorp’s comment reminded me of this:

How to find memory leaks in a Meteor app?

I’ve been having out of memory issues on production since yesterday on my Heroku servers (which run on Amazon EC2). Here’s a completely different tack - try adjusting nodejs’s garbage collection limit? node --max_old_space_size=920 --gc_interval=100 main.js Apparently node’s garbage collection is very lazy, and by default assumes a 1.5GB limit. So on smaller boxes like a micro instance it might be problem. I know it’s for Heroku, but I think it might apply to you. See here: https://devce…

May be worth investigating.

[xunga]

My website crashed every 2.5 days with a CPU going over 100%

Exact the same issue here!

In the end I tried a more expensive package, with more RAM.

I have 8gb of RAM , but are you sure that’s the solution? I’m afraid that It might bite you back in the future

[xunga]

Thanks! I will have a look into it I will keep this post updated…

[xunga]

I’ve profiled the CPU with Kadira and I think my problem probably could be related to aldeed:simple-schema

Stay tuned :'D

[cskaandorp]

I don’t know for sure. So far so good. I kind of regret my decision to do it in Meteor anyway and I am tired of finding solutions for all kinds of problems. The 1(!)Gb package that I have right now can be cancelled at any time, so as soon as I get the same problem, I ditch it. I feel your pain! Good luck man.

[mz103]

Yikes, if aldeed:simple-schema is causing those problems, I’m screwed. I use that for my application.