[Solved] Meteor observe not working properly in 1.6

[petr24]

I recently upgraded from 1.2 to Meteors latest version 1.6.0.1.

I was using observe in a publication and an observe on the client to get changes.

in 1.2 no problems at all, but in 1.6 observed changes are not received in a “changed” client callback, but the client does get the ddp message. I can verify that by looking in Chromes dev tools > websocket, see the incoming message, but it’s never fired in a client callback. This only happens when changing 2-3 documents at a time.

So when I delete a few documents from the DB, the publication fires off the callbacks, and the client receives them in the websocket messages, but it only fires once in the “observe” callback on the client.

Here is my code.

Client -

CollectionTest = new Meteor.Collection('collectionTest');


CollectionTest.find({}).observe({
   added: function (doc) {
      console.log("ADDED DOC ", doc);
   },
   changed: function (newDoc, oldDoc) {
      console.log("CHANGED DOC new ", newDoc);
   },
   removed: function (doc) {
      console.log("REMOVED DOC ", doc);
   }
});

Server -

Meteor.startup(function () {
    Meteor.setInterval(function() {
        console.log("INSERTING DATA ");
        TestData.insert({number: Math.floor(Math.random() * 1000)});
    }, 5000);
});

Server Publication -

Meteor.publish("ddpPub", function () {
    
    var self  = this,
        ready = false;
    
    var userId = self.userId;
    
    var subHandle = TestData.find({}).observeChanges({
        added: function (id, fields) {
            if (ready) {
                self.changed("collectionTest", userId, {
                    type: "added",
                    data: {
                        id: id,
                        fields: fields
                    }
                });
            }
        },
        changed: function (id, fields) {
            if (ready) {
                self.changed("collectionTest", userId, {
                    type: "changed",
                    data: {
                        id: id,
                        fields: fields
                    }
                });
            }
        },
        removed: function (id) {
            if (ready) {
                self.changed("collectionTest", userId, {
                    type: "removed",
                    data: id
                });
            }
        }
    });
    
    self.added("collectionTest", userId);
    self.ready();
    ready = true;
    
    self.onStop(function () {
        subHandle.stop();
    });
});

Attached are images from me removing the documents from the DB. The websocket messages, and then my console on the client. Showing it only fires once for 5 documents.

[petr24]

UPDATE: 12/15/17 - 7:17pm PST

After working on this for a couple hours, finding some related meteor posts with observe callbacks and “Meteor.call” not working inside, the solution or hack is to wrap the “Meteor.call” in a “setTimeout” with the value of 0, and it fixes it.

I tried that here, and it didn’t work, but then I tried throttle the response, and it works!
I am not sure why this works, or what causes the problem in the first place, any explanation would be welcome.

Server Publication -

Meteor.publish("ddpPub", function () {
    
    var self  = this,
        ready = false;
    
    var userId = self.userId;
    
    var subHandle = TestData.find({}).observeChanges({
        added: function (id, fields) {
            if (ready) {
                console.log("ADDING PUBLICATION");
                self.changed("collectionTest", userId, {
                    type: "added",
                    data: {
                        id: id,
                        fields: fields
                    }
                });
            }
        },
        changed: function (id, fields) {
            if (ready) {
                console.log("CHANGING PUBLICATION");
                self.changed("collectionTest", userId, {
                    type: "changed",
                    data: {
                        id: id,
                        fields: fields
                    }
                });
            }
        },
        removed: function (id) {
            if (ready) {
                console.log("REMOVING PUBLICATION");
                ratePub(id, function (data) {
                    console.log("OBJECT DATA IS ", data);
                    self.changed("collectionTest", userId, data);
                });
            }
        }
    });
    
    self.added("collectionTest", userId);
    self.ready();
    ready = true;
    
    self.onStop(function () {
        subHandle.stop();
    });
});

var returnPub = function (id, callback) {
    console.log("RETURNING PUB ");
    callback({
        id: id,
        type: "removed",
        data: id
    });
};

var ratePub = _.rateLimit(returnPub, 10);

[gaurav7]

@petr24, all 3 observe handlers on the server are calling self.changed – is that intentional? With this code, I expect only CHANGED DOC new {...} to be printed on client-side. For the other client-side handlers to be called, the server-side handlers should be calling self.added and self.removed

[petr24]

@gaurav7 I should’ve cleared that up. So in my actual use case, I want to observe users data from a collection, and when a document gets changed in that collection, I fire off a self.changed and send it up to the client. The only reason I need to do it this way vs using Meteors built in reactivity of just doing a Collection.find().fetch() and having the Blaze template take care of it, I am using a JS library to display paginated data, and the JS library doesn’t allow pagination if it’s rendered via HTML, has to be rendered via JS. So I use the observe to be efficient when applying changes. If a user deletes a single document, I don’t need to re-render the entire dataset, I can use my client side observe to remove a single document. Also I am using changed vs removed for removed documents because the remove callback gets called anytime user changes the pagination i.e page, sort. So It’s tough to use for deleted documents from db, when it’s called anytime the subscription dataset is changed.

[gaurav7]

Ahh, so you’re expecting only the changed client-side handler to be invoked, but with different values of the type and data fields.

I wonder if your observation is related to the batching of DDP callbacks introduced in 1.3.3. Some related posts describing the feature:

1.3.3 beta w/ DDP Batching now available!

Hey all, We’ve just pushed a beta release including a larger change that will land in 1.3.3: DDP batching. To upgrade to the release, run: meteor update --release 1.3.3-ddp-batching-beta.0 From History.md: DDP callbacks are now batched on the client side. This means that after a DDP message arrives, the local DDP client will batch changes for a minimum of 5ms (configurable via bufferedWritesInterval) and a maximum of 500ms (configurable via bufferedWritesMaxAge) before calling any callback…

DDP batching on server-side minimongo?

I’m unclear where this ended up. Is there DDP batching on minimongo server-side, and if so, is there a way to disable it? My question relates to server-server streaming and some performance issues that seem to have arisen. Not sure at all that the problem is related to DDP batching but the best way to test that is to turn it off… [client-side its… Meteor.connection._bufferedWritesInterval = 0; ] tks.

My theory is that since the _id is the same for all the changed callbacks, Meteor is now “optimizing” the callbacks and calling only the last one. That makes sense for the default usage of the changed events (e.g. when a cursor is returned from the publication), but seems to be breaking for your unusual usage. To verify the hypothesis, try setting Meteor.connection._bufferedWritesInterval = 0 before subscribing to disable the batching completely?

[petr24]

@gaurav7 HA, awesome! It works. Added it to the client, and everything works like it did before.

I really wish things like this would be mentioned as a breaking change on Meteors site when upgrading to 1.3.3. Have it somewhere in the “Migrating to Meteor 1.x”.

Anyways thanks for the help!!

[gaurav7]

@petr24 Nice! I’m glad you were able to fix it. Though I think your implementation is a bit non-idiomatic; you may wanna consider refactoring to avoid such surprises.

PS: could you please mark it as solved (e.g. add a [Solved] prefix in the title), so others facing similar problems know there’s a possible solution waiting for them here