A company I am using for MongoDB hosting has had a major outage. They know about it (of course) and are working on getting things running. Meantime, since my app can’t connect to their MongoDB cluster, Meteor won’t start up and my app is totally broken. Heroku is giving a totally generic “Application Error” to my users - and when I look in the logs, during restart I see something like this:
2022-06-06T22:38:55.272027+00:00 heroku[web.1]: State changed from crashed to starting
2022-06-06T22:38:55.068977+00:00 app[web.1]: /app/.meteor/heroku_build/app/programs/server/node_modules/fibers/future.js:313
2022-06-06T22:38:55.068997+00:00 app[web.1]: throw(ex);
2022-06-06T22:38:55.068997+00:00 app[web.1]: ^
2022-06-06T22:38:55.068997+00:00 app[web.1]:
2022-06-06T22:38:55.068998+00:00 app[web.1]: MongoError: not master and slaveOk=false
2022-06-06T22:38:55.069002+00:00 app[web.1]: at Connection.<anonymous> (/app/.meteor/heroku_build/app/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:451:61)
2022-06-06T22:38:55.069002+00:00 app[web.1]: at Connection.emit (events.js:315:20)
2022-06-06T22:38:55.069002+00:00 app[web.1]: at processMessage (/app/.meteor/heroku_build/app/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:452:10)
2022-06-06T22:38:55.069003+00:00 app[web.1]: at Socket.<anonymous> (/app/.meteor/heroku_build/app/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:621:15)
2022-06-06T22:38:55.069003+00:00 app[web.1]: at Socket.emit (events.js:315:20)
2022-06-06T22:38:55.069003+00:00 app[web.1]: at addChunk (_stream_readable.js:295:12)
2022-06-06T22:38:55.069004+00:00 app[web.1]: at readableAddChunk (_stream_readable.js:271:9)
2022-06-06T22:38:55.069004+00:00 app[web.1]: at Socket.Readable.push (_stream_readable.js:212:10)
2022-06-06T22:38:55.069004+00:00 app[web.1]: at TCP.onStreamRead (internal/stream_base_commons.js:186:23) {
The error was slightly different earlier in the day (as the provider works to fix things I presume), but I want to know what I can do to trap for DB connection errors and show my users a more useful (and branded) error message that comes from my Meteor app. None of the code in the error trail is mine. We don’t get as far as Meteor.startup() here (or when I intentionally try and force my local dev app to connect to a database that doesn’t exist) so I’m not sure what to do.
I can’t find anything like this on the forum or stack overflow, but I can’t believe I’m the only person who has wanted to do this…
I’m running Meteor 1.11 BTW - and will be upgrading soon.
Personally, I run Meteor behind nginx anyway so would have nginx serve a static error page. Could do similar with Cloudflare or something. Not sure what’s possible on Heroku
That said, I don’t have any handling for early fatal errors like this. I would also be a bit screwed
Guess that’s another ticket for the backlog
This will probably not help you in this case as you’re running on Heroku, but for anyone that has their deployment set up as such that the domain points to an IP (so using MUP, for example), an assignable IP such as Elastic IP on AWS is one solution. If something happens, you can reassign that IP to another app or a simple page and once the main app is back in order, you can immediately revert the IP back to it. This is, of course, manual. But easy to set up.
Change the /etc/mongodb.conf to the path of your mongodb.conf because it varies according to the Linux distribution you are running.
You database has crashed because when you have an outage on Mongo, that’s what it does. You have to repair the database before you can start the service. And make sure the service isn’t running before running repair or it wont do jack.
This is just the way mongo is, it’s caught me and my friends out a few times. And it’s always when you least expect it on a live service and the pressure is on.
I hope this brings it back online soon. If you are still pulling your hair out with it, you can write me here and I will do my best to help. You can also ask in the IRC channels for Mongo and your Linux distribution for support there.
Thanks for the advice @truedon. The mongodb cluster appears to be back on line again (according to their status page), although I am still getting this same error message in my app. I have logged another ticket with the provider and presumably they will know how to do this kind of repair. Since I don’t have access to their mongo instances sitting somewhere in the cloud I doubt I can run it myself.
Have you verified that you can do a direct connection from your local computer to the mongoDB? If so, perhaps during the downtime the DB has been upgraded without your knowledge. Try variations of the connection url, for example for me I had to add ?directConnection=true to get mine to startup.
Turns out I can connect to the instance and read the data in the database using a direct connection and GUI tool (Studio 3T). However, I get the same error master/slave error message in my GUI tool when I try to access the Functions or Users in the database. I am not talking about my own users collection, but the Users object in Mongo that I presume handles mongo security. Presumably heroku/meteor is trying to access this Users thing somewhere in my application code, and that isn’t working properly. Maybe I should try to reset the password or something… I’ll keep digging. Thanks for the tip @james
give it a go in mongo shell rather than in gui, you may uncover hidden behavior not represented by the gui. the users you see in the sidebar of mongo shell is referring to the users property of a database object and afaik meteor does not attempt to read it but only accesses the collections themselves.
meteor also opens up two connections to the mongodb and if you set it up like i do, the users for both connections are different, meaning you should give it a shot trying to connect as the application user and the oplog user (should be read-only) from mongo shell.
I am really struggling to run meteor mongo command on heroku because I just can’t seem to find whereabouts heroku is putting the meteor tool. It may not even do it… I think it just runs node main.js - to actually run meteor. When I run that in the command line, I see the exact same startup errors - with the master/slave stuff. I’m stumped for now as I’m not sure how to access the mongo console via a node command. Hopefully I’ll have some more inspiration later! Thanks for your pointers.
I have got a CLI for Mongo on my dev box, and am able to use it to connect to the production mongo database on my provider. I can connect and query collections fine. This page How to Query MongoDB Secondary Replica Set Members | Studio 3T suggests that my mongoDB instance is no longer set up for secondary reads, and I need to issue rs.secondaryOK() or rs.slaveOK(). Doing that within my mongo client CLI runs fine, but doesn’t seem to make any difference to meteor. Maybe this call needs to happen with meteor somewhere? Running stuff like rs.status() (to see the status of the replica set) or db.system.users.find().pretty() (as suggested by @james) doesn’t work as my user doesn’t have sufficient permission in the database.
Yes, my MONGO_URL env var is set correctly.
I am going to send another support request to my database hosters.
Hope this helps, if not you can try asking your provider for shell access which really should be given if you are paying, if you don’t have shell you can’t do much.
Right on brother, if you don’t got it. You can just setup your own. If you get stuck, dm me. Mongo is free software released open source. It shouldn’t cost anything but the hardware and electric to run it on. Cheers
It turns out that the hoster had changed the connection URI as part of fixing the outage. So my MONGO_URL was set incorrectly. Now that I’ve updated it to the new version, it all works fine. Thanks again for your help.
