Server CPU spikes from oplog

[Dan]

I’m having oplog scaling issues. Every 10 minutes I have a lot of mongodb updates and server cpu spikes. I can’t find much info about what to do about it. I remember seeing some slides from a talk but can’t find them. Anyone have links to info about what people have done to get around too many updates to the oplog.

It’s like before my updates I want to turn off oplog tailing for certain collections then when the updates are done poll to catch up the resume oplog tailing. I saw that 1.3 has options to make a publish function poll instead of oplog tail but it doesn’t look like it’s possible to change that while the app is running.

[benja]

Read these threads.

OPLOG flooding experience thread

Over the past month, I’ve seen and heard of a number of issues where oplog flooding became a problem in production. I’m sure MDG will provide an excellent solution, but as I gear up to launch my next app its becoming my top concern. I thought it may be helpful if people can share their experience so other’s can learn from it and know when to anticipate it. For example: number of db operations / second type of application server configuration type of silver bullets used etc

Oplog tailing too far behind not helping

Hello - having an issue with our application where the CPU usage will keep spiking out of control and cause our app to 499 or 502 until it has recovered itself. At first we had found that were pushing many operations into the oplog and we thought that was bogging down our system, so we set METEOR_OPLOG_TOO_FAR_BEHIND from 2000 to 250, and finally all the down to 100. As this went on, we saw a little bit of alleviation of the spikes, yet they were still happening pretty frequently over the cour…

When there are a large number of updates to the DB the meteor app falls back to polling. This is because oplog tailing is flooding the meteor app which in extreme cases can crash the app. To reduce the CPU spikes you can try to set METEOR_OPLOG_TOO_FAR_BEHIND to a much smaller value but apparently that does not always work.
The other option is to move any collections that requires a large number of batch updates into a separate DB instance, but if you do that you will lose the ability to run reactive queries.

[Dan]

I set METEOR_OPLOG_TOO_FAR_BEHIND to 200. The spikes are still there. I think it’s a little better. Moving the collections to a separate DB isn’t possible.

It seems odd that every server has to watch the oplog or do poll/diff. Why not define one server to do it then send the results to the others.

Also I wish I could control when to pause tailing the oplog then when to do a poll/diff and resume oplog tailing.

It’s impossible to scale unless I’m missing something. Makes all this reactivity useless.

[mz103]

Good post, why delete it?

[sotarules]

mz103 what happened is that just after I posted that, I found Meteor 1.3 parameter disableOplog. I went into my application and selectively applied that to all find functions for the high-insert collection, particularly those instructions in Meteor.publish functions. I’ve been doing some testing, and it is working as advertised. There is still a small CPU spike on the front end Node.js server on the batch insert function, but it is tiny by comparison with what was happening before.

So my deleted post was asking for a robust solution, and in my case it appears that disableOplog is delivers that solution. It allowed me to keep the OPLOG tailing on.

[evolross]

Glad this post wasn’t deleted! This just helped me solve some issues I’ve had for months. Thanks for posting.

[wildhart]

If the spikes are exactly every 10 minutes, could they be due to accounts-password:

CPU spikes every 10 min due to accounts-base package

I have a single combined meteor server and MongoDB (I know, but it works for now!), and precicely every 10 minutes I see a CPU spike: [image] It’s not new but unfortunately I don’t know when I started seeing this. I’ve only just got around to investigating. I’ve tracked it down to the following code in accounts-base/accounts_server.js: const setExpireTokensInterval = accounts => { accounts.expireTokenInterval = Meteor.setInterval(() => { accounts._expireTokens(); accounts._expireP…

[lamhieu]

I a moment, Meteor have recevied many notifications oplog, I think it makes spikes!

[lamhieu]

For this problem, I think Meteor core should update to use stream notifications instead of oplog notifications to reduce notifications we don’t need to receive it!