Big, breaking surprise when copy/pasting documents in MongoChef originally created by Meteor


#1

Hello,

Just wanted to share something I experienced only a few minutes ago.

Using MongoChef, I copied documents and pasted them into the same collection.
I noticed that the _id fields were somewhat different, but didn’t really worry.

The existing _ids were like “Xjgx8G2Kiqvhh7hc2”, and the new ones (for the pasted documents) were like "5816fec9532f72fd0eaa13b7"
Turns out, this breaks everything, as one of my users alerted me while he was lecturing to 200+ students.

The Meteor-created _id fields are string, while the Mongo or MeteorChef created _id fields are ObjectIds.
Any code that does document._id will of course break.

I’m not trying to put blame anywhere, but I’ve been using Meteor now for a long time (2+ years).
This caught me by total surprise, even though I try to read as much documentation as possible.

I hope this post helps someone avoid this issue.

Can someone tell where I can and could’ve learned more about this issue?
Thanks! :slight_smile:


#2

This appears as a MongoChef “feature” to replace ids when creating new documents. Should be easy to change in the settings.


#3

Thanks, daveTheOnly!

However, this occurred to me too, but I could not find a setting in MongoChef for OS X.

Do you know how?


#4

Sorry it is not in the settings… It should prompt you, when you copy. Just tried it with some sample data and it worked without overwriting the ids.


#5

I’m not sure what you mean by overwriting the ids.

If I copy document-a created by meteor, and paste it back in (without deleting document-a first), then it doesn’t ask me anything but “paste documents from clipboard?”

Then, it created document-b with id = ObjectId(). Now, I have two documents: document-a with id=String, and document-b with id = ObjectId().

Is this what you did too?


#6

I copied the document between databases.


#7

Could you try copying the same document into the same collection in the same database and see what happens?
Thanks :slight_smile:


#8

hm, compose.io database wont let me do that…


#9

I’m using Compose.io too, but managing via MongoChef. What error do you get?


#10

Target Database Error msg: not master


#11

I’m guessing your connection has either replicaSet or user with opLog access enabled. True?


#12

You may find the following informative
https://docs.mongodb.com/v3.2/reference/bson-types/#objectid

Mongod will default to ObjectIds if no other _id is specified. I believe it’s also the default for the pure node Mongo driver (Meteor’s Mongo driver wrapper defaults to string _ids, but I believe an option exists to use objectIds instead.)

That said, ObjectIds come in handy – I use them to seed string ids for bulk operations in several heavy backend processes, e.g.

import {ObjectID} from 'mongodb'
doc._id = ObjectID().toString()

#13

I would recommend writing a script to perform any sort of non-trivial database manipulation instead of using Mongochef / other DB GUI built-in functions.

Also, I’d also recommend testing everything in a lower environment before manipulating your production database. That’s where a tested script comes in handy :slight_smile:


#14

Thanks for chiming in, @mds4m!

I agree with your precautions on database manipulation.

This is the first time we’ve had the need to do “copy/move” documents for this purpose, and time was too short for writing code.

I would really like to know more about your seed technique for bulk operations, as I’m doing a lot of bulk operations in the backend. Could you expand upon your use of objectID().toString()?

Thanks :slight_smile:


#15

It’s useful in the Mongo shell, where you can write something like

var newId = new ObjectId().str;

If you’re bulk copying docs for instance.

For regular backend tasks, it comes in handy when you’d like to perform bulk inserts since (to my knowledge) that API is not exposed / implemented on the Meteor Mongo driver. A caveat: any significant write velocities will also manifest in your db’s oplog, so if you’re tailing it for pub/sub you may run into some server resource complications.