MongoDB: Import production DB into dev environment - how can I retain the original document ids?

[cstrat]

So I have a pretty decent amount of data in production, rather than continue with my made up data I want to import the prod data into my dev environment so I can really test out new features.

I’ve pulled a mongodump from the server and have all the data on my dev machine.

However when I do a mongorestore on my development machine it rebuilds the database via inserts - and as such it doesn’t retain the correct _id fields. The result is that all the documents have the right data but get new _ids… so all of my references are broken.

Question: Is there a better way to do this, or am I doing it incorrectly?

[cstrat]

I am guessing that this might not be something people do?
However it would seem that this same issue would occur if you pushed the other way (from dev to prod).

[robfallows]

I was really hoping someone would step in with a definitive answer to this, because according to the documentation for mongodump and mongorestore it should work as you want it to. And, given that relationships based on _ids may exist between collections, it absolutely must retain _ids across a dump/restore.

Are you importing into an empty database in dev (e.g. use meteor reset to clear it out)?

[cstrat]

I was really hoping someone would step in with a definitive answer to this

Yeah me too!
Just would be good to know because it really would help me in my situation.

Are you importing into an empty database in dev (e.g. use meteor reset to clear it out)?

Yup!
Meteor reset the whole thing first.

[pete]

I’m really glad that you posted your question, I have not yet come to a point with my app to have production data but at some point in the next couple of weeks I will and that’s exactly what I was planning to do: download production data to dev and continue work. I was thinking mongo restore would behave much like mysql does when restoring from a backup ane I am hoping there is an explanation to why it did not in your case

[awatson1978]

Meteor Cookbook may have some useful info:
Database Management for Meteor/Mongo Apps

Your mongodump/mongorestore should be preserving _ids. If it’s not, something isn’t configured correctly. Are you dumping to .bson data? If you’ve walked around the bson, and are using the json file options, that could cause a reparse of the data.

If none the mongodump/mongorestore recipes in the cookbook help, try using the db.copyDatabase() function. It’s what to use in production devops situations, where you want to hot-copy entire databases from one running instance to another.

db.copyDatabase('staging', 'meteor', 'localhost');

The only caveat with db.copyDatabase() is that it expects itself to be run on port 27017. Since the meteor database runs on port 3001, you won’t be able to copy from your production database to it. Rather, you’ll need to set up a separate mongodb instance and then point your app’s MONGO_URL environment variable to it.

[cstrat]

hey thanks for the reply!!

All I did was run mongodump on the server.
It provided bson files for the data, and json files for the metadata.
On my dev machine - I ran mongo reset and then started the environment to get mongo running.

Once all running, I initiated the restore with: mongorestore -h 127.0.0.1 --port 3001 -d meteor .meteor/prod_dump/
Obviously the mongo dump was put in that folder: .meteor/prod_dump/

It looks like I ran pretty much the same command you ran as documented in the cookbook, except the flags I used are a little different (-d rather than --db).

[shock]

good point, any tips how to do redirect in that case?
It should be possible with iptables, or some other tool, maybe even netcat.
As nobody want to pipe it trough SSL what would be standard communication channel when copying over unsecured network with SSH tunnel.

[marklynch]

Mongorestore is used to restore backups so it should be identical. I use the nightly backup of my production DB to troubleshoot problems on my local machine a lot. Here’s the command line I use to restore it. The --drop makes sure it clears the collections before adding to them.

mongorestore --host 127.0.0.1 --port 3001 --db meteor path-to-your-dump/your-db-name --drop

Hope that helps

[drone1]

This was useful. Thanks!