Minimongo bulk update? Update in a loop is very very slow when there's an observer running

help
1

I’ve a Meteor collection that’s client only. Its data is coming from a method call, and is not reactive. I’m managing the minimongo collection like this:

Meteor.call('getData', null, function(error, documents) {
  for (var doc of documents) {
    ClientCollection._collection.update(
      {_id: doc._id},
      doc,
      {upsert:true}
    );
  }
});

The collection is not particularly big (~300 documents) and that works ok (~50ms)

Now if I have an observer running when the update is being done, this becomes extremely slow (~5s) (My observer returns the n top results. The sort might make things worst) My guess is that the observer will update an each insert, and rerun things on components that depend on this observer. Sounds very bad.

I’ve tried to wrap this loop between ClientCollection._collection.pauseObservers(); and ClientCollection._collection.resumeObservers(); to no avail.

I am right thinking the pointer is rerunning at every iteration of the loop? I’ve not found a way to do bulk update. Is there one? What’s the best way to handle this? Thanks!

1 Like
2

Unfortunately, I think this is not possible.

Feel free to check out the source:

3

It is the only way. Meteor subscribe does the same when changes come from server. [quote=“gsabran, post:1, topic:18964”]
this becomes extremely slow (~5s)
[/quote]
Try to fetch plain results without sort or limit. Cursors on the client side isnt the best place for doing that.

4

Ok thanks

That what I guessed. Too bad because it’s very nice to use the same db oriented API syntax to get data. Would it be faster to fetch plain result and sort/filter in normal javascript?

5

Yes it would be faster. If you look into Minimongo implementation youll see it is a cascade of underscore (_) functions. And it is much more faster if you apply all filters and sorting by the hand on an entire collection data. Unfortunatelly(

Can you share your observers code? May be it could be optimized or made be non-reactive/

6

Cool I’ll ding into minimongo code (hope I would not have to do it!) From what I see, it’s a bit weird that it’s not delivering super well on the implicit promise that it’d be like a db, on the client, and that we can just talk to it like we would to mongo. It’s surprising that it’s supporting APIs like sorting but that it’s still better to do it by hand. But then I know I’m not using it exactly as excepted since I’m using a client side only collection. Anyway thanks a lot for the insights.

Let me wrap up some clean code

Yeah, it’s similarly annoying to have to make things non reactive for them to work fast while we’re using meteor for the easy to use reactivity :slight_smile:

7

Why are people telling you its impossible ? this function will only trigger observer once (on last item) when inserting bulk

export function insertBulk(collection, documents) {
    const last = _.last(documents);
    _.each(documents, item => {
      if (item._id === last._id)) {
        collection.upsert({ _id: last._id}, item);
      } else {
        collection._collection._docs._map[item._id] = item;
      }
    });
}

Full version with ObjectID :

8

No one told that it is impossible. We are about it is very slow to run observers after bulk update

9

the first comment said:

“Unfortunately, I think this is not possible.”

then you said
"[pausing observers] It is the only way. Meteor subscribe does the same when changes come from server.

both are completely false statements.

My function only triggers observer on last item regardless how many docs you are inserting

10

Oh, man I dont really want to play with words. Anyway, let’s imagine that I used your code

const Test = new Mongo.Collection(null);
Test.find({age: {$gte: 10}}).observe({
  added: (doc) => {
    console.log(doc)
});

bulkInsert([{_id: "1", age: 11}, {_id: "2", age: 14}]);
// after that only {_id: "2", age: 14} would trigger observer
11

Do the upserted documents prior to the last go through the observer?

12

Using the method mentioned by @sikanx they dont come in observer callbacks. You should recreate a Cursor/Observer after insertBulk to see the changes. The magic is that meteor saves the state of collection into Collection._collection._savedOriginals every insert/update/remove and then diff _savedOriginals and current _docs._idmap to find changes and pass to Cursors. And if you change _docs._idmap directly it doesnt come into diff.

13

are you sure ? did you actually try my function or just imagined that you did ? The upsert call on the last item is a regular call will make sure that all changes are reflected.

14

Dear @sikanx it is not a competition. The tone of your replies is not friendly and I suppose you want show us that you are expert, but is you?


insertBulk = (collection, documents) ->
  if collection
    last = _.last(documents)
    _.each documents, (item) ->
      if _.isObject(item)
        console.log 'bulked'
        if item._id is last._id
          collection.upsert { _id: last._id }, item
        else
          if _.isObject(item._id)
            string = item._id.toHexString()
            collection._collection._docs._map[string] = item
          else
            collection._collection._docs._map[item._id] = item

Test = new Mongo.Collection(null)

docs = _.map _.range(500), (index) ->
  return {_id: index.toString(), index}

Cursor = Test.find({})
Cursor.observe
  added: ->
    console.log 'added'
  changed: ->
    console.log 'changed'

Meteor.startup ->
  console.log 'insert bulk'
  insertBulk Test, docs
  Meteor.defer ->
    console.log 'upsert minimongo way'
    _.each docs, (doc) ->
      Test.upsert({_id: doc._id}, {index: doc._index})


# Output:
# insert bulk
# 500 times bulked
# 1 time added
# upsert minimongo way
# 499 times added
# 1 time changed

Take a note we talk about observer. I did try your code. It works as you mentioned. But it doesnt work with observer. More over if you try to upsert the same docs after insertBulk, you may see that it triggers added again, because insertBulk do not care about the diff. But anyway Cursor.fetch() returns you correct idmap = an Array of those 500 docs without any issue and you can render them!

P.S. I hope you can be more polite, because here we try to help each other and no one wants to take a crown

2 Likes