Blocking collection updates in a loop

help
1

Hi guys

I need to loop over an object, and upsert for each key value pair in the object.

Currently I’m doing something like

for (key in obj) {
  Collection.upsert(query, update);
}
//Logs after loop has finished
console.log('Done')

However, I have read in the docs Collection.upsert blocks on the server.
I am calling upsert ~7000 times. Does upsert block in this situation? Should i be doing something more like

const upsertPromises = [];
for (key in obj) {
  Collection.upsert(query, update, (err, changed) => {
    if (err) return reject(err);
    resolve(changed);
  });
}
Promise.all(upsertPromises)
.then(() => console.log('Done'))
.catch((err) => console.log(err));

Essentially, I’m checking whether the db updates are called in parallel, or sequence when in a loop.

Thanks :smiley:

2

Essentially, you’re correct in that MongoDB operations are done sequentially. However, you may find this post interesting if you’re interested in performance:

As far as using Promises is concerned, it would be better to make use of the underlying rawCollection() methods, as they’re Promisified already:

const upsertPromises = Objecct.keys(obj).map(key => {
  return Collection.rawCollection().update(query, update, { upsert: true });
});
Promise.all(upsertPromises) ...
2 Likes
3

Thanks for that link. Really insightful article, shame i couldn’t find it through my googling.

The article mentioned collection.rawCollection should be used for batch operations of any type.

Does your snippet demonstrate the correct usage?

4

Should be correct, but the update syntax for an upsert is from memory.

5

Use Mongo Bulk API for best efficiency.

Here’s the gist (untested)

const bulk = Collection.rawCollection().initializeUnorderedBulkOp()
for (...) {
  bulk.find(query).upsert().updateOne(updates)
}
Meteor.wrapAsync(bulk.execute, bulk)()

https://docs.mongodb.com/manual/reference/method/Bulk.find.upsert/

1 Like
6

That’s really helpful. Thanks :smiley:

7

Following up on this.

I tried to return Collection.rawCollection().update(query, update, { upsert: true }); in a ValidatedMethod method and got an Error: max call stack size exceeded. Collection.rawCollection().update(query, update, { upsert: true }); itself seems fine as i am able to add a then and catch chain without trouble inside the method. The error only occurs when i return the Collection.rawCollection().update(query, update, { upsert: true }); promise.

Additionally, the promise shown below returned (and worked) fine (based on my original pattern) with exactly the same query and options as the erroneous one:

Collection.upsert(query, update, options, (err, changed) => {
  if (err) return reject(err);
  resolve(changed);
});

Any idea why this may be?