MongoDB query in MeteorJS takes 20-40 seconds to execute. I have a MongoDB collection of 10million+ documents of OHLCV price. Find query takes 0.2 seconds, but when I use fetch() in MeteorJS it increases to 20-40 seconds. The fields in the query are indexed. I can’t figure out what is causing the fetch() function to be so slow.
When you run find() - the node driver just registers a cursor with the DB, it does very little work (almost none in fact). Calling .fetch() (known as toArray() in the node driver documentation) actually pulls the data into the server - it as at this point the query is actually evaluated against the server, so the cost of .find() should almost always be close to 0. The cost of .fetch() will be a combination of:
the query evaluation time (e.g., figuring out which documents match)
serializing the data on the mongo server
de-serializing the data on meteor
The cost of (1) probably isn’t optimized in your current setup. In general, mongo will use exactly 1 index to evaluate your query and will try to “guess” which is the best, it’s not always great at guessing. In your situation you probably want a compound index: { tickerFormatted: 1, timeframe: 1, start: 1 } - this will mean your query is entirely covered by your index.
If your documents are very large outside of the fields you need - you can also add the projection fields to the index to make a covered query - this is one where mongo doesn’t even need to fetch the document (from memory or more likely disk) to serve the query - however it does make your index very large, so use carefully. There are also caveats when used in meteor (specifically you must exclude _id from the projection).
These things are most easily debuggable from the mongo shell with find(query).explain() - which will tell you which index to use - if you want to know how many documents will be returned you can use explain({ verbosity: "executionStates" }) - this is where you’ll see how well your indexes cover your query.
For (2/3) - the issue will be that calling fetch() pulls in all the data at once. If you don’t need it all at once, use .forEach() - this will not reduce the execution time, but it will reduce memory usage. The counter-point to this is that it must not take you more than 10 minutes to process each batch of data or the cursor (the result of .find()) will timeout.