Oplog errors with Meteor app in Azure and mongo hosted at Compose

jocatalin · August 12, 2015, 4:20pm

Hi,

I’m running a dockerized meteor app (meteord image) via Tutum.co ( a platform as a service) using Microsoft Azure for the infrastructure. My Mongo db is running in Compose.io
The Meteor app is connected to the Mongo DB via the env variable:
MONGO_URL mongodb://user:pass@lamppost.11.mongolayer.com:10199,lamppost.10.mongolayer.com:10217/testdb?replicaSet=set-5593a9752debaf23ed000477
And this works fine.

If I try to add the METEOR_OPLOG_URL env variable with the value

mongodb://user:pass@lamppost.11.mongolayer.com:10199,lamppost.10.mongolayer.com:10217/local?authSource=testdb

I’m getting errors in the meteor app.

The user has oplog access.
The database is hosted by Compose.io in AWS Ireland zone. The errors are happening with the meteor app in Azure datacenters (I tried Amsterdam and Ireland).
I added the meteor app in AWS Ireland, it worked from there. (without changing anything in the env variables for METEOR) Also running the app locally (via docker) works OK! But we want to use Azure. Anybody else running in this config? Azure + Compose?

kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z Exception in setInterval callback: Error: failed to connect to [lamppost.11.mongolayer.com:10199]
kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z     at Object.Future.wait (/built_app/programs/server/node_modules/fibers/future.js:398:15)
kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z     at [object Object].<anonymous> (packages/meteor/helpers.js:119:1)
kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z     at [object Object].MongoConnection.(anonymous function) [as remove] (packages/mongo/mongo_driver.js:678:1)
kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z     at [object Object].Mongo.Collection.(anonymous function) [as remove] (packages/mongo/collection.js:575:1)
kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z     at _cleanStaleResults (packages/oauth/pending_credentials.js:30:1)
kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z     at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z     at packages/meteor/timers.js:6:1
kompas-test-web-4 | 2015-08-12T15:28:02.438069500Z     at runWithEnvironment (packages/meteor/dynamics_nodejs.js:108:1)

kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z Error: No primary found in set
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at Object.Future.wait (/built_app/programs/server/node_modules/fibers/future.js:398:15)
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at new MongoConnection (packages/mongo/mongo_driver.js:213:1)
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at new MongoInternals.RemoteCollectionDriver (packages/mongo/remote_collection_driver.js:4:1)
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at Object.<anonymous> (packages/mongo/remote_collection_driver.js:38:1)
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at Object.defaultRemoteCollectionDriver (packages/underscore/underscore.js:750:1)
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at new Mongo.Collection (packages/mongo/collection.js:98:1)
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at packages/accounts-base/accounts_common.js:141:1
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at /built_app/programs/server/packages/accounts-base.js:243:4
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at /built_app/programs/server/packages/accounts-base.js:1659:3
kompas-test-web-3 | 2015-08-12T15:20:19.988757700Z     at /built_app/programs/server/boot.js:222:10

kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z Error: No valid replicaset instance servers found
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at Object.Future.wait (/built_app/programs/server/node_modules/fibers/future.js:398:15)
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at new MongoConnection (packages/mongo/mongo_driver.js:213:1)
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at new MongoInternals.RemoteCollectionDriver (packages/mongo/remote_collection_driver.js:4:1)
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at Object.<anonymous> (packages/mongo/remote_collection_driver.js:38:1)
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at Object.defaultRemoteCollectionDriver (packages/underscore/underscore.js:750:1)
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at new Mongo.Collection (packages/mongo/collection.js:98:1)
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at packages/accounts-base/accounts_common.js:141:1
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at /built_app/programs/server/packages/accounts-base.js:243:4
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at /built_app/programs/server/packages/accounts-base.js:1659:3
kompas-test-web-6 | 2015-08-12T15:20:26.778887200Z     at /built_app/programs/server/boot.js:222:10

johanbuys · September 18, 2015, 7:37am

Exact same issue. Azure using compose.io.

Exception in setInterval callback: Error: failed to connect to [lamppost.xx.mongolayer.com:10049]

Works fine on local using same db and connection.

@jocatalin do you perhaps use mup to deploy?

Were you able to resolve this issue?

shock · September 18, 2015, 7:47am

I would check if there is no firewalling issue from Azure to amppost.xx.mongolayer.com:10049

using some command line tools like

telnet amppost.xx.mongolayer.com 10049
nc amppost.xx.mongolayer.com 10049

If there is no connection check firewall.

And from speed/traffic point of view, I would really suggest using Azure Compose when you want to have application on Azure.

johanbuys · September 18, 2015, 8:59am

The behaviour is very strange and sporadic, after a successful login everything works and you can continue working for a while, but then it breaks and the logs spit out can’t connect. After the connection issues you can’t login and everything related to the DB seems to be non functional. As if the connection was just dropped without reason and is not able to reconnect.

I am running a pure node app on another azure instance using mongoose and Compose.io almost the same issues were occurring and support from compose suggested that I enable “keep alive” in mongoose and that fixed the problem. So this might be related?

mac0 · November 21, 2015, 11:30am

I have the exact same problem. I can’t figure a way to get rid off this. Any luck on your side?

mac0 · November 23, 2015, 12:52pm

I finally found the solution! It has to do with Azure having some kind of idle time policy.
Adding a few options to my mongo url connection did the trick:

"MONGO_URL": "mongodb://username:password@url1:27018,url2:27019/dbName?replicaSet=replicaName&readPreference=primaryPreferred&w=majority&connectTimeoutMS=60000&socketTimeoutMS=60000

PS: adding options for readPreference and majority also solved meteor connection problem when primary is down and replicaSet elects new primary.
Hope this helps!

shwaydogg · December 14, 2015, 9:44pm

Thanks for sharing your findings! Is

Did you leave the oplog url alone or were there updates there as well?

I’ve added the following to the end of my MONGO_URL as provided by compose are there any other changes to make?

&readPreference=primaryPreferred&w=majority&connectTimeoutMS=60000&socketTimeoutMS=60000

Seems to have solved the Exception in setInterval callback but some things are still taking a very long time such as logging out.

Sidenote the mongoDB reference for url options is: https://docs.mongodb.org/v3.0/reference/connection-string/#uri.connectTimeoutMS

mac0 · December 15, 2015, 8:36am

Hey,

I don’t seem to have problem of any kind anymore. Will check carefully if I observe a strange behavior on logout.

As per Oplog, I changed it as well with those options:

local?authSource=admin&replicaSet=mongors&connectTimeoutMS=60000&socketTimeoutMS=60000

It seems to work like a charm even though I am not sure all of those options are required.

evolross · April 6, 2017, 7:12am

Sorry to awaken this, but I’m getting this error on Galaxy in my app. Seems related. I’m going to try the above tip with adding the options to the mongo URL. Will report back the results. I’m on Meteor 1.2.1. Mongo is hosted at compose.io Also, in the line replicaSet=mongors in the above Oplog URL, should that be set to the same value as the Mongo URL replicaSet?

2017-04-05 15:39:23-07:00Exception in setInterval callback: Error: failed to connect to [c1005.candidate.64.mongolayer.com:11005]
jtg7
2017-04-05 15:39:23-07:00 at Object.Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:398:15)
jtg7
2017-04-05 15:39:23-07:00 at [object Object]._.extend._nextObject (packages/mongo/packages/mongo.js:994:1)
jtg7
2017-04-05 15:39:23-07:00 at [object Object]._.extend.forEach (packages/mongo/packages/mongo.js:1028:1)
jtg7
2017-04-05 15:39:23-07:00 at [object Object]._.extend.map (packages/mongo/packages/mongo.js:1038:1)
jtg7
2017-04-05 15:39:23-07:00 at [object Object]._.extend.fetch (packages/mongo/packages/mongo.js:1062:1)
jtg7
2017-04-05 15:39:23-07:00 at [object Object].Cursor.(anonymous function) (packages/mongo/packages/mongo.js:877:1)
jtg7
2017-04-05 15:39:23-07:00 at [object Object].cursorProto.(anonymous function) (packages/meteorhacks_kadira/packages/meteorhacks_kadira.js:2996:1)
jtg7
2017-04-05 15:39:23-07:00 at [object Object].kadira_Cursor_fetch [as fetch] (packages/meteorhacks_kadira/packages/meteorhacks_kadira.js:3377:1)
jtg7
2017-04-05 15:39:23-07:00 at after (packages/matb33_collection-hooks/packages/matb33_collection-hooks.js:369:1)
jtg7
2017-04-05 15:39:23-07:00 at Object.fields (packages/matb33_collection-hooks/packages/matb33_collection-hooks.js:392:1)
jtg7
2017-04-05 15:39:23-07:00 - - - - -
jtg7
2017-04-05 15:39:23-07:00 at [object Object].<anonymous> (/app/bundle/programs/server/npm/npm-mongo/node_modules/mongodb/lib/mongodb/connection/server.js:556:74)
jtg7
2017-04-05 15:39:23-07:00 at [object Object].emit (events.js:106:17)
jtg7
2017-04-05 15:39:23-07:00 at [object Object].<anonymous> (/app/bundle/programs/server/npm/npm-mongo/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:156:15)
jtg7
2017-04-05 15:39:23-07:00 at [object Object].emit (events.js:98:17)
jtg7
2017-04-05 15:39:23-07:00 at Socket.<anonymous> (/app/bundle/programs/server/npm/npm-mongo/node_modules/mongodb/lib/mongodb/connection/connection.js:534:10)
jtg7
2017-04-05 15:39:23-07:00 at Socket.emit (events.js:95:17)
jtg7
2017-04-05 15:39:23-07:00 at net.js:441:14
jtg7
2017-04-05 15:39:23-07:00 at process._tickDomainCallback (node.js:492:13)