Meteor Range Error out of memory

help
1

Hello,

SO im using python to do some data aggregation and computationally complex tasks. the result is a file that is about 300mb and has about 125,000 lines.

Using pymongo, I push this database into meteor. and get this, almost every time.

Exception in defer callback: RangeError: Out of memory
I20160817-15:14:16.079(-4)?     at RangeError (native)
I20160817-15:14:16.079(-4)?     at DocFetcher.fetch (packages/mongo/doc_fetcher.js:60:8)
I20160817-15:14:16.079(-4)?     at packages/mongo/oplog_observe_driver.js:509:43
I20160817-15:14:16.079(-4)?     at [object Object]._.extend.forEach (packages/id-map/id-map.js:48:1)
I20160817-15:14:16.080(-4)?     at packages/mongo/oplog_observe_driver.js:507:35
I20160817-15:14:16.080(-4)?     at packages/mongo/oplog_observe_driver.js:16:9
I20160817-15:14:16.080(-4)?     at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
I20160817-15:14:16.080(-4)?     at packages/meteor/timers.js:6:1
I20160817-15:14:16.080(-4)?     at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1)

This suddenly started happening after weeks of the app running just fine.

Each client is only ever subscribed to 500 ish documents at a time.

My theory is that this happens because meteor is reading each entry as a change, and trying to push each database change to the client, rather than something in bulk?

Is there anything I can do? Would this problem go away if this were not on the development server and the server had more ram.

2

I had that happen to me when a collection was missing an index.

3

What do you mean by index? Currently Im using the randomly generated _Id provided by mongo

4

You should look for the queries you are running in your code and pass them through the query analyzer. It will tell you what mongodb is doing.

If you are querying the database for a particular field that has no index associated with it, I believe mongodb driver needs to load the entire collection which results in the error you see. That is at least consistent with my observations.

5

my queries are simple find statements with one or two parameters, the other has a few $regex and $in statements… could that contribute to the problem? Thanks for the help by the way.

6

Of course your queries are find statements. Which fields are your querying? Do you have an index on the fields you are querying?

7

No, i dont think so ill look into using indices thanks!