Steadily increasing CPU % on Meteor v1.10.1

Hey guys

Any idea how to troubleshoot/solve this issue I’m facing… The CPU on my production node(s) increases steadily over a matter of weeks. Any tips appreciated. I have started using node Events recently (but also upgraded Meteor from 1.8 {the latest version [now using Node 12] does seem to need more RAM & CPU for sure} and changed a few other packages etc.) so cannot pinpoint the issue easily as it takes weeks to manifest.

The dip around the ~26th March was a deployment, which resets CPU usage back to zero for a while.

Thanks

Interesting I have never seen a CPU usage increase steadily like that…this looks like a memory graph…do have any scheduled jobs? Or perhaps some computation that increases linearly with data?

I have daily batch jobs that trigger once a day. Using the synced-cron package, but these are not new…

How do I profile what’s happening? Using meteor-up to deploy, and can’t find any info on how to enable profiling.

I have started using node Events

How do you actually used them and does the spike also occur when you take an older snapshot without new features but upgraded Meteor to 1.10.1?

I came here just to open the same discussion… This is a standby web application. 2 users doing minimal user tests once in a while. The drop was indeed a server restart. No scheduled jobs or any code difference from 1.8.3.

I posted this in the past: Memory Utilization is running high It looks like those “saw teeth” are now adding up like … a desire which is not being released and keeps accumulating in various parts of the body … if I may …

2 Likes

Very interesting and very similar. Good to know you had no code changes…

I guess we need to profile and see what is using those CPU cycles. Doing this in a non obtrusive way (when the issue only manifests over weeks) is the tricky part.

Can profiling be enabled dynamically? To run in production with profiling permanently is not feasible…

I did find this issue which covers the use case quite well: https://github.com/nodejs/diagnostics/issues/148

This looks promising too https://medium.com/voodoo-engineering/node-js-and-cpu-profiling-on-production-in-real-time-without-downtime-d6e62af173e2

I think the best way to start is to survey the community and see if many see the same pattern.

I am having a similar problem. Since I’ve upgraded to 1.10.1 I am observing both CPU and Memory slowly and steadily growing. Here just last week’s view:

Upon first start, node was using just 150 MB of memory.
Over time this is growing up to a point it reached 500MB.
I restarted node, and it’s back to 135MB, but I see it’s again slowly growing memory utilization.

CPU we can also see it’s steadily growing, but since values are slower, the increases are also small, and it takes longer to observe the growth.

This is my PROD environment, and I have an exact same environment, version and code on an INT environment, but there I DO NOT observe this behavior. Here’s the INT graph for the same week:

I’ll be following this thread in case there’s something new - I’ll also post updates if I find something on my side.

1 Like

I’ve seen this many times with Meteor
Likely related to this: Memory Utilization is running high but it happens for CPU as well.

In summary:

  • Never ever use observe/observeChanges in your code. (Imo this should be banned from the Meteor docs and not be publicly available)
  • Avoid publishing data, only publish ids, and grab results with a method
  • Implement https://montiapm.com/ and have a look at what else is creating observers. Could be some package

and you should be fine.

(in other words, avoid using the Meteor ‘black magic’)

1 Like

If you are using Meteor Up, you can run run npx --package mup@next mup meteor debug to use Chrome’s DevTools for Node to take CPU and memory profiles. The feature is part of the 1.5 beta, but it still works when the app was deployed with older versions. Also, Monti APM’s paid plans provide a remote cpu profiler.


I am working on a low overhead continuous profiler service (initially for CPU, but eventually for memory also). If anyone wants to join the private beta when it is ready, you can send me a message.

1 Like

I think it would be ok now to open an issue for this matter so we can track it. At least for me it is clear that 1.8.3 and 1.10.1 have different patterns with the same code and same standby usage.

1 Like
3 Likes

Thanks for opening the issue Paul. I’ve deployed some changes so I can profile on demand in future. I’ll update the ticket if I find anything useful.

It was really easy to generate a profile on demand using the native inspector-api btw.

For people seeing the problem. Can you also mention which is the last version you tested where you didn’t see the problem. And specifically if you saw it on 1.9

Unfortunately I skipped 1.9 due to various factor including ironically Mongo client version. More below.

I switched to a blue/green deployment (with DNS failover) this week and see the same CPU increase over a matter of days across both nodes. The warm node is occasionally used for admin, but no user traffic. So I don’t think it’s a flaw in my app code.

BLUE/hot:

GREEN/warm:
Screenshot 2020-04-24 at 18.51.55

I’ve attached the node js CPU profile (on github issue) I took for 10min on the GREEN/warm host. You can import into Chrome devtools but I’ve included a screenshot below.

My observations:

  • Some timer is running every 10 seconds
  • A lot of the CPU time is spent in the npm-mongo package and also in processTicksAndRejections()

Not experienced analysing cpu profiles though so any input appreciated…

Hi guys, maybe this was related to an issue on npm mongodb driver, it was updated already in the Meteor wrapper as well, read more here https://github.com/meteor/meteor/issues/11026

9 Likes

That’s great to hear! :+1:

This issue is a good example of why having a community of people with more or less similar stack (specailly backend) is a good thing as oppose to a siloed communities for the build system, view layer, data layers etc… The system needs all the part to work well together to be of any use.

NodeJS is a set of many small libraries which is not a bad thing. But maintaining and debugging those integrations is not an easy undertaking. In this incident, we’ve a group of smart people identifying, isolating and fixing the issue for which the rest of the Meteor community is grateful.

12 Likes

So for anyone following…

Upgrading to v1.10.2 (specifically npm-mongo 3.7.1) resolved this for me.

5 Likes