I’ve noticed in my MongoDb database that all of a sudden larger numbers are stored as NumberLong.
Examples:
{ “_id” : “NLxy4Ms6LLMwWA8vF”, “totalGenome” : NumberLong(“4045945861”) }
{ “_id” : “WoRG79GccLTtJMqJd”, “totalGenome” : NumberLong(“3939367540”) }
{ “_id” : “2sgNAsBwK9Ynp8aBj”, “totalGenome” : 3740883272 }
{ “_id” : “aHHJuaHE9Sv6WhMtv”, “totalGenome” : 3666860656 }
{ “_id” : “ZWL9pZ7cvw6AR5KnM”, “totalGenome” : NumberLong(“3603169670”) }
{ “_id” : “fKFs5Zi4XJTvoaqci”, “totalGenome” : 3564251572 }
{ “_id” : “PaKXivADC2LswRZfz”, “totalGenome” : 3549303056 }
{ “_id” : “X9rgFNy8iTjQ2QNjT”, “totalGenome” : NumberLong(“3481553695”) }
According to this SO answer: MongoDB differences between NumberLong and simple Integer? this is coming from the MongoDb driver.
I’m 100% sure we haven’t change any code that alters this relevant number field at all.
The numbers are also calculated wrongly and shouldn’t be so big in the first place.
Has anyone else seen this problem? If you app handles large numbers in fields, please check before you see strange behavior because of this bug.
Does that mean that your app is still calculating the numbers correctly, but they just somehow get bigger while inserting or updating into MongoDB?
The numbers are incorrect, to quite some extent. See this example:
That’s 12.3 x more than it should be (tgSize is the correct value vs the other one coming from the doc itself).
Can you observe the exact number in a debugger session just before the mongo insert or update takes place? It is so hard to believe that mongo makes such a huge mistake and this is the first time that it pops up.
Don’t you by any chance use Collection2 or any other package that automatically transforms some values before making an insert or an update?
We’re using the raw MongoDb collection since over a year as we needed Transactions feature. So far it was without any problems but this problem has been for at least 2 months (based on some of the updatedAt fields).
I haven’t been able to isolate it yet as it’s happening on Production 
Added some consoles to see which function is producing this exactly.
No, we just use the check() functions to ensure that parameters are correct.
Only other thing is this little piece of code on inserts/updates:
const Kits = new Mongo.Collection('kits');
// Collection hooks to add created and updatedAt automatically
Kits.before.insert(function (userId, docParam) {
const doc = docParam;
doc.created = new Date();
});
Kits.before.update(function (userId, doc, fieldNames, modifierParam) {
const modifier = modifierParam;
modifier.$set = modifier.$set || {};
modifier.$set.updatedAt = new Date();
});
const KitsRaw = Kits.rawCollection();
That’s a mysterious error. You could maybe find the earliest occurrence of this phenomenon in your database, and check the changes in git just before that date. I would probably go back in time 1…3 deployments (app versions) prior to the earliest finding, because if I understand correctly, the error does not occur consistently, so it may be that the code change goes back to more than just one version.
I know this is not much of a help, but this is what I would do.
Another approach would be to force activities in development resembling those in production hoping that the error occurs, and once it does, you can pinpoint it.
Does anyone have an idea how I can identify those docs with NumberLong in the totalGenome field?
At least then I can correct it more efficiently, we have almost 100k docs in this collection and in order to calculate I need to query another collection with almost 7k docs.
I’m a bit puzzled about how those NumberLong values made it into your database in the first place. As it turns out, NumberLong is neither a type in Javascript, nor in MongoDB as such, except in the mongo shell, see shell types in mongodb, NumberLong.
My current understanding is that there is no other way in getting a value of such type into MongoDB other than via the console. A NumberLong…
is to handle 64-bit integers.
Both Javascript and MongoDB treat numbers as 64-bit floats:
The JavaScript Number type is a double-precision 64-bit binary format IEEE 754 value, like double in Java or C#.
…and the same seems to apply to MongoDB too. So even if you set an integer larger than 32 bits as a value in MongoDB, it won’t be somehow converted into a NumberLong; instead it will be set just like a regular Javascript Number, as above.
That raises the question if there is any workflow in your system that uses the mongo shell for data insertion or manipulation. At this moment that seems to be the only plausible way of how these values got into your database.
Apparently there is no way to query documents in MongoDB where the type of a particular field would be NumberLong, since, as stated above, this is for some reason not even a data type in MongoDB – it’s just a shell function. However, NumberLong values are stored as 64 bit integers with a particular type designator, so such documents could theoretically be parsed from a MongoDB bson dump.
See also https://stackoverflow.com/q/17185220
You should be able to do .find({ totalGenome: { $type: 18 } }) - if that doesn’t work, check for the different types here: $type — MongoDB Manual
Edit: just read @peterfkruger reply - so perhaps I’m wrong
2 Likes
You’re not wrong at all – thanks for finding this, I couldn’t figure it out myself.
Many thanks for that. I could identify 107 docs (out of almost 10k) that had this problem and fix it.
No idea what caused it (as Peter has highlighted it can only be done by running a shell script) but it’s fixed now and I will keep a close eye on it should it ever turn up again.
1 Like