Bulk edit database

I’m working on arabic-russian dictionary. I have 40000 documents in Articles collection. Each article contains from words, translations and examples.

words: [ { note, word } ]
translations: [
    {
        translation, 
        examples: [ ], 
        tags: [ ], 
        images: [ ]
    }
], 
morphologies: [ ]

Now I want to add additional fields to each article documents:
wordsCount, translationsCount, examplesCount, morphologiesCount, and maybe count length of all text fields…

How should I do this? I have read some docs and watched youtube tutorials about bulkWrite, Bulk.find.update. They teach to manually create new data, and send them to server at once. And my question is:

  1. should I prepare large object with 40000 entries, with each of my updates?
  2. From where should I run this code? I need some button in client interface, fires Bulk.execute?

When you are changing the schema of a collection, you are performing a migration. There are several atmosphere packages to help you do this (there may also be some NPM packages).

Each migration is normally a one-time activity which is run on the server.

1 Like

Here is a blueprint for using bulkWrite:

import { Meteor } from 'meteor/meteor';

Meteor.methods({
    migrateStuff() {
        const ops = MyCollection.find({}, { ... }).map((document) => {
            return {
                updateOne: {
                    filter: { _id: document._id },
                    update: { $set: { someField: someValue } },
                }
            };
        });
        MyCollection.rawCollection().bulkWrite(ops, { ordered: false });
    },
});
1 Like

Thanks. It is exactly what I was seeking!
And it will work without problems for calculations in loop, and 40000 entries?

If it works depends on factors on the data you throw at the server and how big your server and maybe the bandwitdth is.

const bulkSize = 500;
const runs = Math.ceil(count / bulkSize);
for (let i = 0; i < runs; i++) {
    const documents = collection.find(query, { skip: i * bulkSize, limit: bulkSize });
    Meteor.call('migrateStuff', documents);
}

I wouldn’t do it exactly with a method call, this is just to build on the other example, but it should give you an idea how to slice this in batches to not overload your server. You could also pause (Meteor.setTimeout) between each batch as the database call will return but the db is still working. You will have to experiment a little.

2 Likes