Transferring to a remote database troubles (formerly Meteor 1.2.1 and Mongo 3.2.4 - not compatible?)

help
1

I’ve been working to shift the databases of my apps from Meteor’s internal Mongo to an external Mongo.

This is all on my local Mac right now.

Originally, MongoDB 2.4.6 was installed and used by Meteor to run it’s internal instance of Mongo.

I added an installation of MongoDB 3.2.4 using brew. I switch back and forth between the two using brew switch mongodb #.#.#.

First I did a mongodump from the original internal database (2.4.6).

Next I switched to mongodb 3.2.4 and did a mongorestore. Seemed to work fine. Could find all the data using the mongo shell.

Started meteor with MONGO_URL=‘mongodb://localhost:27017/’ meteor.

Started up fine. App works, sort of. The accounts packages can’t find any users, even though they are all there. The other collections return no documents or many fewer documents in them than are really there.

I’m thinking there’s an incompatibility here somewhere.

Any ideas?

2

Check if by any chance string ids (Meteor standard) weren’t replaced by objectIds (Mongo standard).

3

I’ve tried using 3.2.0 and it seems to work fine with users.

4

The ids look the same in the db.___.find() returns. Don’t know what’s happening inside Mongo, though. It changed storage to WiredTiger during the transition and I’m wondering if that’s part of the problem.

5

Did you migrate from an internal meteor mongo instance or did you start from scratch?

6

I’m not using Mongo 3.2 myself, but I do remember a Compose.io article that Meteor is not compatible with 3.2 yet.

Here’s the link - scroll to “Meteor Oplog Issue” section:

https://www.compose.io/articles/connecting-to-the-oplog-on-the-new-mongodb/

7

tyujupov - Thanks for that. I’m just now sitting down to remove that instance of mongo and try earlier instances. Will keep this thread posted on my results.

8

Update – I uninstalled MongoDB 3.2.4 and reinstalled MongoDB 2.4.12. I then ran the mongorestore into this new instance. All the collections and documents seem to be there based on my examination using the mongo shell, but again, the Meteor app cannot access many of them. No users, only subsets of or no posts. The app can access all the documents just fine in the meteor internal mongo database. What’s the difference?

I’m considering using mongoexport and mongoimport and/or db.copyDatabase(), but the Mongo docs seem to highly prefer the mongodump and mongorestore method. Will these methods make any difference, since I can’t see anything wrong with the data from the dump/restore process?

Surely (not Shirley) other people begin their development and implementation using the internal database and later move to an external database? How do they do it?

9

I tried to to that once but kept having issues where mongorestore was creating documents with new _ids, so none of my data was useful. The data wasn’t that important to me so I wasn’t too upset scrapping it and just rebuilding it on the prod server.

10

I’m not getting new _ids per the mongo shells, comparing the same query results for the internal DB and the external DB, but I’m wondering about brajt’s comment “Check if by any chance string ids (Meteor standard) weren’t replaced by objectIds (Mongo standard).”

I can’t find a way to discover whether the ids are objects or strings. They all look the same in the query results. I tried looking for a parameter to add to the mongodump and/or mongorestore, but haven’t found any.

This is a serious problem for us. We’ve got several of these nascent sites in use. We’re ready to scale up, but we can’t start over. There is important data in those internal databases, including media in gridFS collections.

Ouch!

11

One of the things I have discovered is about the accounts package. While the users database contains all the user docs, the app can’t find them.

I even created a new user in the remote database using the Join capabilities of the accounts-ui. I signed in successfully using the new user, but “if(Meteor.user())” returns false all the time, even when the new user is signed in. Also the new user does not appear in the users database, even though it continues to sign me in successfully.

So, if I can’t create a new user in the remote database that is recognized by Meteor.user(), then there must be something more basic wrong here.

I’ve created other records (posts) and they come up fine, user or no in the remote database, but most of the existing posts do not (only 14 out of 121).

I need to rename this topic. It’s more basic than Meteor 1.2.1 vs MongoDB 3.2.4. It more clearly about working with a remote database, especially one that has data copied in.

12

AFAIK, Meteor does not officially support MongoDB 3.x series yet.

Meteor and MongoDB work differently on _id fields, which may very well cause missing or not-working-as-expected records. Meteor saves them as plain strings, MongoDB saves them as ObjectId()s. Mongodump and mongorestore may be getting confused by string _ids.

Have you tried writing a script to export the records from within Meteor with Mongo 2.x and then importing them from within Meteor with Mongo 3.x? I suspect this will help with the missing or not-working records.

Also, apparently mongodump does not dump the local database. Not sure if this changes anything, but why not.

There is also the --oplog option. I suspect it wouldn’t help you but no harm in trying.

And also there is the possibility of copying over the files in /data/db directly. If the file formats are compatible, this should solve your problem.

13

necmettin – Thanks for a couple of good leads.

I can imagine that the local database might be part of the answer to the Meteor.users() problem. I’ll check that out.

I’m thinking now that the _ids might be the key to everything. Thanks for using the word “confused” in your speculation. That explains things better, I think, because the app thinks some records are there and some are not. It’s not all or nothing. Although everything is there in the mongo shell.

I don’t think the MongoDB version issue is going to end up being as significant as it appears right now. I am having the same exact problems when I restore to the same version of Mongo as meteor is using.

I’ll post what I find in the morning.

Bye the bye. The files are not the same between Mongo 2.4 and Mongo 3.x. The WiredTiger storage engine does things completely differently at the file level.

14

After much experimentation and observation I can now more clearly describe what is happening here.

An internal database is created in a meteor app. Data is added, both through a fixture script and through normal use.

A mongodump is done.

A mongorestore is made to a mongod instance, this one is on 127.0.0.1:27017.

At this point, if you look at both databases using mongo shells, the are identical, the internal meteor database and the new meteor database on the mongod instance.

Fire up the meteor app using – MONGO_URL=“mongodb://localhost:27017/” meteor

As meteor connects to the mongod instance, a new “admin” database is created by mongo. This database is an exact copy of the meteor database, except that all the _ids have been recalculated. None of the original _ids exist here.

The app begins running on the admin database with all the new _ids, not the meteor database as expected. Of course these new _ids break connections all over the app, but also, only a subset of the data is written to the admin database. Only 14 of what were 18 records, for instance.

I would/am considering not using _ids to connect anything or find anything or for anything at all. In many developments before I’ve used a separate UUID, apart from the database row identifier. This way, if you have to change databases, the complication of maintaining the original row based id is alleviated. So, I’m not averse to doing this here, but it won’t solve 2 problems. First, I don’t have an easy way to add new fields to the user database (a UUID). I’m using the standard accounts packages and I’d rather not have to modify those packages. Second, only a subset of the data is being written to this admin database. And, of course, why is the app running on the admin database and not the meteor database?

I would appreciate any insight into this. Surely this cannot be the way that meteor intended? Can it?

Thanks in advance, Alex

15

Meteor 1.2.1 uses a version 1.4.39 mongodb node driver, which is not compatible with MongoDB 3.2.x. It is compatible with 3.0.x though. Right now I am using 3.0.10 in my production setup and everything is gravy. Here is the official compatibility chart:
https://docs.mongodb.org/ecosystem/drivers/driver-compatibility-reference

I know there is an open issue right now about upgrading the driver to 2.1 and there is a lot of support for that to happen.

16

You must give database name, not just server URL: MONGO_URL="mongodb://localhost:27017/_dbname_" meteor run

It is possible that not giving the database name is the sole reason for all your troubles.

I know you’re looking for a way out, but that is a very, very bad solution. _id is the primary key, you would have to rewrite practically everything.

Let’s see what happens after you give the database name.

1 Like
17

necmettin – Thank you so much. The database name was the key to the whole thing. Everything works fine when the name is included properly.

As to the separate UUID, I chose not use that approach when I began this series of projects because mongo uses it’s own form of UUID in its _id. Unique across platforms and data moves. Glad to see that works.

Best, Alex

18

Glad your problems are solved. Too bad it took us this long to figure it out. This was a learning experience for me too.

19

This last successful dump/restore was to/from MongoDB 2.4.12. Now I’m going to try to restore to MongoDB 3.x. I want to deploy apps on a shared node server that is running MongoDB WiredTiger, so I need to see how high I can go in version numbers. The server is newly setup and is running MongoDB 3.2, but I think it can be set back to 3.0 if need be. From what danryan said, I expect 3.0 to be the max right now.

20

I’ve taken the time to test a mature app running Meteor 1.2.1 against an external MongoDB 3.2.4 instance using WiredTiger. I populated the database using dump files from a MongoDB 2.6 instance. I used mongorestore into the the new 3.2.4/WiredTiger instance.

Then I ran the app against it adding new content, editing old content. It was a thorough test including some large media files in gridfs.

I haven’t found anything amiss, haven’t been able to break anything.

My understanding was that this latest version of MongoDB (3.2.4) wouldn’t work. Or is it just that it isn’t officially supported? What kind of problems should I be looking for?