I deployed a meteor app to a aws server using mup which used docker to deploy the app. I also used Mlab sandboxed db. The issue is, the app crash without warning after some time. Crashing means the app still work but data won’t load from the database. I have two versions of the app (staging and production) running in two different servers. But both have this issue. In the docker logs I see this error repeated a lot of times :
Exception in setInterval callback: MongoError: server instance pool was destroyed
at Object.Future.wait (/bundle/bundle/programs/server/node_modules/fibers/future.js:446:16)
at MongoConnection.<anonymous> (packages/meteor/helpers.js:119:1)
at MongoConnection.(anonymous function) [as update] (packages/mongo/mongo_driver.js:771:49)
at [object Object].update (packages/mongo/collection.js:589:29)
at AccountsServer.Ap._expireTokens (packages/accounts-base/accounts_server.js:1100:14)
at packages/accounts-base/accounts_server.js:1173:14
at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
at packages/meteor/timers.js:6:1
at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1)
- - - - -
at Function.MongoError.create (/bundle/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:29:11)
at basicWriteValidations (/bundle/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/server.js:433:51)
at [object Object].Server.update (/bundle/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/server.js:543:16)
at [object Object].Server.update (/bundle/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/server.js:386:17)
at updateDocuments (/bundle/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:1037:19)
at [object Object].Collection.update (/bundle/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:1075:44)
at MongoConnection._update (packages/mongo/mongo_driver.js:560:18)
at MongoConnection.<anonymous> (packages/meteor/helpers.js:118:1)
at MongoConnection.(anonymous function) [as update] (packages/mongo/mongo_driver.js:771:49)
at [object Object].update (packages/mongo/collection.js:589:29)
The crash was it a ‘Server Crash - write after end’ by any chance?.. I’m seeing something similar in my logs, also no idea why, the system is not being used at the time of a crash.
In the client browser console, I see this error WebSocket connection to 'wss://oursite.com/sockjs/373/2lrpal10/websocket' failed: WebSocket is closed before the connection is established.
Exception in setInterval callback: MongoError: server instance pool was destroyed at Object.Future.wait (D:\home\site\wwwroot\programs\server\node_modules\fibers\future.js:446:16) at MongoConnection. (packages/meteor/helpers.js:119:1) at MongoConnection.(anonymous function) [as update] (packages/mongo/mongo_driver.js:771:49) at [object Object].update (packages/mongo/collection.js:589:29) at AccountsServer.Ap.expireTokens (packages/accounts-base/accounts_server.js:1100:14) at packages/accounts-base/accounts_server.js:1173:14 at [object Object]..extend.withValue (packages/meteor/dynamics_nodejs.js:56:1) at packages/meteor/timers.js:6:1 at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1) - - - - - at Function.MongoError.create (D:\home\site\wwwroot\programs\server\npm\node_modules\meteor\npm-mongo\node_modules\mongodb-core\lib\error.js:29:11) at basicWriteValidations (D:\home\site\wwwroot\programs\server\npm\node_modules\meteor\npm-mongo\node_modules\mongodb-core\lib\topologies\server.js:434:51) at [object Object].Server.update (D:\home\site\wwwroot\programs\server\npm\node_modules\meteor\npm-mongo\node_modules\mongodb-core\lib\topologies\server.js:545:16) at [object Object].Server.update (D:\home\site\wwwroot\programs\server\npm\node_modules\meteor\npm-mongo\node_modules\mongodb\lib\server.js:386:17) at updateDocuments (D:\home\site\wwwroot\programs\server\npm\node_modules\meteor\npm-mongo\node_modules\mongodb\lib\collection.js:1041:19) at [object Object].Collection.update (D:\home\site\wwwroot\programs\server\npm\node_modules\meteor\npm-mongo\node_modules\mongodb\lib\collection.js:1079:44) at MongoConnection._update (packages/mongo/mongo_driver.js:560:18) at MongoConnection. (packages/meteor/helpers.js:118:1) at MongoConnection.(anonymous function) [as update] (packages/mongo/mongo_driver.js:771:49) at [object Object].update (packages/mongo/collection.js:589:29) at AccountsServer.Ap.expireTokens (packages/accounts-base/accounts_server.js:1100:14) at packages/accounts-base/accounts_server.js:1173:14 at [object Object]..extend.withValue (packages/meteor/dynamics_nodejs.js:56:1) at packages/meteor/timers.js:6:1 at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1)
severityLevel
1
I am seeing these issues in our failed request routing:
We just ran into this issue on Meteor Galaxy, also updated to the new Meteor version earlier this week. The bug started coming up for the first time today. Our database is provided by a Sandbox cluster on MongoLabs, which doesn’t provide any redundancy – is there a chance this is happening because they temporarily took down our database? It’s certainly up now and the app is stuck in this crash cycle, not sure what to do to get up and running again.
Help!!
EDIT: We’re also users of Kadira, and our stack trace includes references to one of its collections – maybe that’s connected?
MongoError: server instance pool was destroyed
at Object.Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:446:16)
at MongoConnection.<anonymous> (packages/meteor/helpers.js:119:1)
at MongoConnection.(anonymous function) (packages/mongo/mongo_driver.js:771:49)
at [object Object].remove (packages/mongo/collection.js:624:29)
at _cleanStaleResults (packages/oauth/pending_credentials.js:30:1)
at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
at packages/meteor/timers.js:6:1
at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1)
- - - - -
at Function.MongoError.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:29:11)
at basicWriteValidations (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/server.js:433:51)
at [object Object].Server.remove (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/server.js:574:16)
at [object Object].Server.remove (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/server.js:393:17)
at removeDocuments (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:1227:19)
at [object Object].Collection.remove (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:1267:44)
at MongoConnection._remove (packages/mongo/mongo_driver.js:417:16)
at MongoConnection.kadira_MongoConnection_remove [as _remove] (packages/meteorhacks_kadira/lib/hijack/set_labels.js:29:1)
at MongoConnection.<anonymous> (packages/meteor/helpers.js:118:1)
at MongoConnection.(anonymous function) (packages/mongo/mongo_driver.js:771:49)
EDIT 2: After going back through the logs, it looks like the first exception started when our MongoDB set couldn’t be reached. We got a bunch of versions of the error below on different queries from different collections, every error appearing at the same moment, then started getting the “server instance pool was destroyed” error about 30 seconds later (and every minute since then). Anyone got any ideas? Is this a consequence of using the MLab Sandbox on production? Is there a way to rescue the app? The database is now accessible, I’ve checked from the terminal.
9nje
2016-09-28 16:48:51-04:00Exception while polling query {"collectionName":"users","selector":{"_id":{"$in":["RhDbqYQ7x8W4h27zT","iaXx5esvQ5hQi4v97","mtanBuMzEmnkYN47N","kgQZfkQiZJuJcEsQc"]}},"options":{"transform":null}}: MongoError: failed to reconnect after 30 attempts with interval 1000 ms
9nje
2016-09-28 16:48:51-04:00 at Object.Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:449:15)
9nje
2016-09-28 16:48:51-04:00 at SynchronousCursor._nextObject (packages/mongo/mongo_driver.js:1021:47)
9nje
2016-09-28 16:48:51-04:00 at SynchronousCursor.forEach (packages/mongo/mongo_driver.js:1055:22)
9nje
2016-09-28 16:48:51-04:00 at SynchronousCursor.getRawObjects (packages/mongo/mongo_driver.js:1104:12)
9nje
2016-09-28 16:48:51-04:00 at PollingObserveDriver._pollMongo (packages/mongo/polling_observe_driver.js:152:48)
9nje
2016-09-28 16:48:51-04:00 at PollingObserveDriver.proto._pollMongo (packages/meteorhacks_kadira/lib/hijack/wrap_observers.js:100:1)
9nje
2016-09-28 16:48:51-04:00 at Object.task (packages/mongo/polling_observe_driver.js:90:12)
9nje
2016-09-28 16:48:51-04:00 at [object Object]._.extend._run (packages/meteor/fiber_helpers.js:147:1)
9nje
2016-09-28 16:48:51-04:00 at packages/meteor/fiber_helpers.js:125:1
9nje
2016-09-28 16:48:51-04:00 - - - - -
9nje
2016-09-28 16:48:51-04:00 at [object Object].<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:290:15)
9nje
2016-09-28 16:48:51-04:00 at emitTwo (events.js:87:13)
9nje
2016-09-28 16:48:51-04:00 at [object Object].emit (events.js:172:7)
9nje
2016-09-28 16:48:51-04:00 at Socket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:160:49)
9nje
2016-09-28 16:48:51-04:00 at Socket.g (events.js:260:16)
9nje
2016-09-28 16:48:51-04:00 at emitOne (events.js:77:13)
9nje
2016-09-28 16:48:51-04:00 at Socket.emit (events.js:169:7)
9nje
2016-09-28 16:48:51-04:00 at emitErrorNT (net.js:1269:8)
9nje
2016-09-28 16:48:51-04:00 at nextTickCallbackWith2Args (node.js:442:9)
9nje
2016-09-28 16:48:51-04:00 at process._tickCallback (node.js:356:17)
If you are seeing this issue “MongoError: server instance pool was destroyed” can you please post the following information there:
Mongo setup (are you using a DBaaS, if so which type/plan, if not what is your replicaset setup?)
Which version of the mongo driver you have (which version of the npm-mongo package).
Hopefully a pattern will emerge and we can get a reproduction of this issue going, which will make it possible trace the problem to the mongo driver or Meteor’s wrapper.
@tmeasday I am going to come clean here: Just remembered we are using a logging adapter that consumes all console statements and sends them to our ApplicationInsights tracer / exception logs.
So what we maybe seeing is just an error, but we do not get to see an indication of recovery.
Having the same problem on local dev from time to time when loosing internet connection. Using mLab sandbox for local (no oplog url) development. I believe it has to do with the autoreconnect in the mongo driver (using 1.4.1.1).
THe mongo_url env variable looks like this
Hi @tmeasday, this began happening to me locally, a few times today - see the screenshot attached.
I’m not sure what initiates the error. I’m running on a Macbook and suddenly the fan would start running like crazy and the whole OS would freeze with exceptions of allowing me to click on my browser and code editor. Any OS menus/taskbar would be inaccessible. I was not able to scroll in terminal and that’s the only screenshot I have for now.
It would be interesting to know if it was mongo or meteor that locked up your CPU… Perhaps you could leave a window running top open somewhere on screen and when things crashed you might see
Exception in setInterval callback: MongoError: server instance pool was destroyed
at Object.Future.wait (/Users/fabien/.meteor/packages/meteor-tool/.1.4.1_2.1jlej5l++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:446:16)
at MongoConnection.<anonymous> (packages/meteor/helpers.js:119:1)
at MongoConnection.(anonymous function) [as remove] (packages/mongo/mongo_driver.js:771:49)
at [object Object].remove (packages/mongo/collection.js:624:29)
at _cleanStaleResults (packages/oauth/pending_credentials.js:30:1)
at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
at packages/meteor/timers.js:6:1
at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1)
- - - - -
at Function.MongoError.create (/Users/fabien/.meteor/packages/npm-mongo/.1.5.50.126i5ny++os+web.browser+web.cordova/npm/node_modules/mongodb-core/lib/error.js:29:11)
at basicWriteValidations (/Users/fabien/.meteor/packages/npm-mongo/.1.5.50.126i5ny++os+web.browser+web.cordova/npm/node_modules/mongodb-core/lib/topologies/server.js:434:51)
at [object Object].Server.remove (/Users/fabien/.meteor/packages/npm-mongo/.1.5.50.126i5ny++os+web.browser+web.cordova/npm/node_modules/mongodb-core/lib/topologies/server.js:578:16)
at [object Object].Server.remove (/Users/fabien/.meteor/packages/npm-mongo/.1.5.50.126i5ny++os+web.browser+web.cordova/npm/node_modules/mongodb/lib/server.js:393:17)
at removeDocuments (/Users/fabien/.meteor/packages/npm-mongo/.1.5.50.126i5ny++os+web.browser+web.cordova/npm/node_modules/mongodb/lib/collection.js:1232:19)
at [object Object].Collection.remove (/Users/fabien/.meteor/packages/npm-mongo/.1.5.50.126i5ny++os+web.browser+web.cordova/npm/node_modules/mongodb/lib/collection.js:1272:44)
at MongoConnection._remove (packages/mongo/mongo_driver.js:417:16)
at MongoConnection.<anonymous> (packages/meteor/helpers.js:118:1)
at MongoConnection.(anonymous function) [as remove] (packages/mongo/mongo_driver.js:771:49)
at [object Object].remove (packages/mongo/collection.js:624:29)
Same Error here. Happening since today. Rolled back to 1.4.1 and it seems to work for now.
