[Solved] Avoid printing error: E11000 duplicate key

Hi, I am trying to add 400k+ documents, which are stored in files, which gives a lot of duplicate key errors so having this error printed in the console makes the process very slow. The duplicate key errors are unavoidable and the try .. catch around insertMany does nothing for me. How can I make the entire insert process happen while not having these error messages printed?

TradesCollection.createIndex( { "startName": 1 }, { unique: true } );

Meteor.startup(() => { 
  const dataFiles = fs.readdirSync(process.cwd().split('.meteor')[0] + './server/data');
  
  dataFiles.forEach(dataFile => { 
    const data = JSON.parse(fs.readFileSync(process.cwd().split('.meteor')[0] + 'server/data/' + dataFile, 'utf8'));

    try {
      TradesCollection.rawCollection().insertMany(data, { ordered: false, forceServerObjectId: true });
    } catch (e) {
        // do nothing
    }
  });
});

rawCollection().insertMany() (and all other calls on rawCollection) are async and return promises - so you’d probably need to .catch(() => {}) the result at minimum, or make your for each callback async, convert it to a map and wrap it in an await Promise.all, or - wrap the call in Promise.await.

This will at the very least help, though I’m not totally sure it will resolve your problems entirely as I’d expect you to get uncaught promise errors that you don’t mention.

You could patch console.error before the call, and un-patch it after (not a great solution I admit)

1 Like

The .catch(() => {}) worked without errors, thank you!

For anything like this I would not have it inside of Meteor at all and simply create a script that can be run outside of Meteor and direct to the database with node mongo driver, its very fast (1m+ inserts in under a minute on my local machine)

1 Like