Mongo Scaling Issues

SUCCESS!

Thanks for the help guys! Pretty sure I just solved the Mongo problems that I was facing.

Here’s a mini blog post on the process I went through:

Quick background: I’m using Compose.io for MongoDB hosting (their MongoDB Classic package as of now). The app was being really slow at responding and I didn’t understand why. Compose.io support were suggesting I add more RAM, which I did, but that really didn’t help anything and was just costing me more money. Kadira was showing me something like this:

(Response time goes back to normal at around 22:08 which is when I solved the problem).

The database was taking ~30s to respond to queries during peak traffic, but I couldn’t understand why. Adding more servers didn’t help anything and actually made things worse because the DB couldn’t handle all the connections as @int64 mentioned above. (I was running 50+ Meteor instances at some points. Now it’s about 20).

Kadira helped by showing me that the method response time was slow due to Mongo taking ~30secs to respond!

But this only helped me part of the way because all DB operations were slow. So how do I find the problem?

Ask on the forums, SO, and Compose.io support :slight_smile: which led me to:

During peak times I logged in to Compose.io and had a look at the current ops that were running and saw something like this (but far worse):

Had no idea what all this mumbo jumbo was, but you’ll see that each op has a secs_running field. In the image above it says 0 seconds for everything, which is great! But what I was seeing during peak time was 14 seconds, 9 seconds, 10 seconds… for the different operations that were going on! And it was all coming from the same query being made by my app.

I ran this query myself and it really did take something like 16 seconds to get a response! Not good! And running it with explain (as @hwillson suggested) showed that 180,000+ documents were being scanned! Here’s an example of a problematic query:

Anyway… lo and behold, there’s no index set up for such a query, so I added teamId1: 1, gameweek: 1 and teamId2: 1, gameweek: 1 as indexes and ran the query again and I get the response in milliseconds.

After this the whole database starts acting quickly again. This one problematic query was slowing down the entire database!

I use indexes for almost every publication, but this is one that I forgot to add and it wasn’t easy to find initially!

Hope this helps someone in the future. Might make it to Medium at some point. Or not :slight_smile:

Thanks a lot for the help again!