Meteor: Why is Oplog so slow?


#1

Hey,
we are running multiple Meteor instances for a live chat application. Since 2 days I’ve noticed some very high CPU usage on our servers, even the pub/sub response times have increased (from about 20ms up to 200ms). After checking the Kadira logs, I’ve found this:

Fetching oplog.rs takes up to 2600ms. This is a very high value, so I’m wondering if there are any ways to improve it? Or should we go back to Meteors polling solution and set a small pollingInterval value (f.e. 1 second)?


#2

We have the same issue when running the latest version of npm-mongo (1.5.48 or newer). I think there is an issue with the latest version of the node mongo driver. We’ve reverted to 1.5.47 for now.


#3

So you are using Meteors mongo@1.1.11 instead of mongo@1.1.12?


#4

Some other Kadira graphic that irritates me is this one:

I could swear that it was full of oplog notifications just 2-3 weeks ago. We didn’t change any of the pubs or subs.


#5

Yes, we are using mongo@1.1.1.


#6

Thanks, just downgraded to 1.1.1 - for the moment it seems like it solves the problem, I’ll keep my eye on it.

@benjamn Is this a known issue, or should we open a new issue on GitHub? Don’t know if we can make an easy reproduction.


#7

Hi,
I’m trying to downgrade my mongo version from 1.1.12 to 1.1.11 by setting it in my .meteor/versions and .meteor/packages files. But everytime I start my app, it upgrade it back to the 1.1.12.

Thanks for help


#8

Did you try it with this command?

meteor remove mongo
meteor add mongo@=1.1.11

#9

No I didn’t, I do it now :slight_smile:


#10

So it doesn’t work. I got a huge message error when I do the adding command :

MINISAT-out: Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 67108864, (2) compile with ALLOW_MEMORY_GROWTH which adjusts the size at runtime but prevents some optimizations, or (3) set Module.TOTAL_MEMORY before the program runs.
MINISAT-err: Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 67108864, (2) compile with ALLOW_MEMORY_GROWTH which adjusts the size at runtime but prevents some optimizations, or (3) set Module.TOTAL_MEMORY before the program runs.

/Users/robin/.meteor/packages/meteor-tool/.1.4.1_1.1l0s2pc++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/isopackets/ddp/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:165
      throw error;
      ^
abort() at Error
    at jsStackTrace (packages/logic-solver/minisat.js:3:1)
    at stackTrace (packages/logic-solver/minisat.js:3:1)
    at abort (packages/logic-solver/minisat.js:33:1)
    at enlargeMemory (packages/logic-solver/minisat.js:3:1)
    at Function.Runtime.dynamicAlloc [as alloc] (packages/logic-solver/minisat.js:3:1)
    at _sbrk (packages/logic-solver/minisat.js:3:1)
    at Sd (packages/logic-solver/minisat.js:7:1)
    at Ud (packages/logic-solver/minisat.js:7:1)
    at gc (packages/logic-solver/minisat.js:7:1)
    at pc (packages/logic-solver/minisat.js:7:1)
    at Kb (packages/logic-solver/minisat.js:7:1)
    at Dc (packages/logic-solver/minisat.js:7:1)
    at Object.dd [as _addClause] (packages/logic-solver/minisat.js:7:1)
    at packages/logic-solver/minisat_wrapper.js:74:1
    at Object._native.savingStack (packages/logic-solver/minisat_wrapper.js:37:1)
    at [object Object].MiniSat.addClause (packages/logic-solver/minisat_wrapper.js:68:1)
    at [object Object].Logic.Solver.solve (packages/logic-solver/logic.js:1326:1)
    at [object Object].Logic.Solver.solveAssuming (packages/logic-solver/logic.js:1369:1)
    at minMaxWS (packages/logic-solver/optimize.js:52:1)
    at [object Object].Logic.Solver.minimizeWeightedSum (packages/logic-solver/optimize.js:139:1)
    at packages/constraint-solver/solver.js:368:1
    at Function.time (/tools/tool-env/profile.js:305:10)
    at [object Object].CS.Solver.minimize (packages/constraint-solver/solver.js:352:1)
    at [object Object].CS.Solver.minimize (packages/constraint-solver/solver.js:346:1)
    at [object Object].CS.Solver._getAnswer (packages/constraint-solver/solver.js:735:1)
    at packages/constraint-solver/solver.js:603:1
    at Function.time (/tools/tool-env/profile.js:305:10)
    at [object Object].CS.Solver.getAnswer (packages/constraint-solver/solver.js:602:1)
    at packages/constraint-solver/constraint-solver.js:165:1
    at Object.Logic.disablingAssertions (packages/logic-solver/logic.js:45:1)
    at Function.CS.PackagesResolver._resolveWithInput (packages/constraint-solver/constraint-solver.js:164:1)
    at [object Object].CS.PackagesResolver.resolve (packages/constraint-solver/constraint-solver.js:128:1)
    at /tools/project-context.js:518:35
    at Function.time (/tools/tool-env/profile.js:305:10)
    at /tools/project-context.js:514:19
    at /tools/utils/buildmessage.js:359:18
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:352:34
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:350:23
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at Object.enterJob (/tools/utils/buildmessage.js:324:26)
    at /tools/project-context.js:486:20
    at /tools/packaging/catalog/catalog.js:100:5
    at /tools/utils/buildmessage.js:271:13
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:264:29
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:262:18
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:253:23
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at Object.capture (/tools/utils/buildmessage.js:252:19)
    at Object.catalog.runAndRetryWithRefreshIfHelpful (/tools/packaging/catalog/catalog.js:99:31)
    at ProjectContext._resolveConstraints (/tools/project-context.js:485:13)
    at /tools/project-context.js:268:9
    at /tools/utils/buildmessage.js:359:18
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:352:34
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:350:23
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at Object.enterJob (/tools/utils/buildmessage.js:324:26)
    at ProjectContext._completeStagesThrough (/tools/project-context.js:258:18)
    at /tools/project-context.js:250:12
    at Function.run (/tools/tool-env/profile.js:489:12)
    at ProjectContext.prepareProjectForBuild (/tools/project-context.js:249:13)
    at /tools/cli/commands-packages.js:2034:20
    at /tools/utils/buildmessage.js:271:13
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:264:29
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:262:18
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at /tools/utils/buildmessage.js:253:23
    at [object Object].withValue (/tools/utils/fiber-helpers.js:89:14)
    at Object.capture (/tools/utils/buildmessage.js:252:19)
    at Command.func (/tools/cli/commands-packages.js:2033:27)
    at /tools/cli/main.js:1410:23
If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information.

#11

Did you try it with the equal? In the first version I forgot this.


Meteor performance bottleneck - EDIT: poor oplog performance
#12

Yes yes I tried exactly that :confused:


#13

Mhh okay, I’m also on 1.4 and Mac, everything here works fine. Maybe you can try a reboot :slightly_frowning_face:


#14

haha ok. I gonna try :slight_smile:
Thx. I come back to you soon :slight_smile:


#15

So I just rebooted, and I still have the same error when doing the meteor add mongo@=1.1.11 :frowning2:


#16

Mhh okay, but going back to

meter add mongo

works without problems?


#17

yes I just tried and works fine


#18

Mh okay, my last idea would be to remove all caches/build in your .meteor folder, f.e.

.meteor/build
.meteor/plugin-cache
.meteor/bundler-cache

and then run the remove/add command again.


#19

still the same :frowning2:


#20

Nevermind I will try to find how to fix that. Thanks a lot for your help.