Overlap when subscriptions with skip/limit and subscriptionsReady


#1

The subject seems a bit obtuse, but I didn’t know how else to describe in a few words what is happening.

I am trying to page through a large collection on the client by virtue of subscribing with skip/limit. It works well, except for the fact that for a short period of time the client has an interleaved document set when subscriptionsReady fires, which results in a flickering display when no longer subscribed documents are removed from the screen.

Lets say we subscribe in our Template

GigaCollection = new Meteor.Collection('gigaCollection');
Template.gigaCollection.onCreated(function(){
 var self = this;
 var skip = new ReactiveVar(0);
 var limit = 12;
 self.autorun(function(){
  self.subscribe('gigaCollection', {sort:{name:1},skip: self.skip.get(), limit: self.limit});
 })
});

and access the documents in via a helper

  Template.gigaCollection.helpers({
   docs: function() {
    return GigaCollection.find({}, {sort: {name:1}});
   }
  });

what I see on the page when changing the skip value is more than 12 items that (depending on network speed) sift down to the 12 items from the subscription, indicating that the local mongo collection seems to have old documents in it when the new documents come from the server and it takes some time for the old ones to be removed, which shows reactively on the screen.

My question: how can I tell that the subscription is really ready and only contains the documents I actually subscribed for?


Overwriting data returned from subscription by query
Incorrect result set in template data context
#2

I am also very interested in this. In my case I see this problem when changing to another route, which does not have the same subscription as the previous route. For a second, the user sees how the “old” documents are being removed from the screen before the new route and template are displayed. Really need a solution for this.


#3

@jamgold: you have just hit one of the most terrible Meteor limitation. See:


#4

And same probleme here:


With still no answer from MDG…


#5

Phew, I am just glad I am not the only one. I have found several workarounds which at first seemed like hacks, but in light of other people having the same issues seem worthwhile sharing.

I will post once I am back at my desk


#6

hey @jamgold, i’ve posted what I believe a related issue. it seems like minimongo’s limit may sometimes not work properly

minimongo-limited-query-initially-returns-all-documents


#7

I have found two different workarounds.

The first does not rely on Template.instance().subscriptionsReady() and instead uses a reactive var to trigger readiness

Template.gigaCollection.onCreated(function(){
 var self = this;
 var ready = new ReactiveVar(false);
 var skip = new ReactiveVar(0);
 var limit = 12;
 self.autorun(function(){
  self.ready.set(false);
  self.subscribe('gigaCollection', {sort:{name:1},skip: self.skip.get(), limit: self.limit},function(){
   self.ready.set(true);
  });
 })
});

this solution really tests for readiness :slight_smile:

The second hack has to do with making sure a Collection.find after a skip/limit shows only the new dataset. It is a bit more involved by adding a value to the result set that is uniq to the request. You can use a uniq id (Meteor.uuid) or a timestamp. I had success by simply adding the skip value.

Collection = new Meteor.Collection('something');
if(Meteor.isServer) {
 Meteor.publish('something', skip, limit, function(){
  var handler = Collection.find({},{skip: skip, limit: limit}).observeChanges({
     added: function(id, doc) {
      doc.skip = skip;
      self.added('something', id, doc);
     }
  });
  self.ready();
  self.onStop(function () {
   if(handler) handler.stop();
  });
 });
}
if(Meteor.isClient) {
 Template.something.onCreated(function(){
  var self = this;
  this.autorun(function(){
   self.subscribe('something', Session.get('skip'), 10);
  });
 });
 Template.sometemplate.helpers({
  something: function() {
   return Collection.find({skip: Session.get('skip')});
  }
 });
}

Let me know what you think


Caching subscription ids in minimongo to vastly simplify client-side querying
#8

Maybe I am missing something, but it looks like the first workaround is equivalent to checking the ready state of the subscription. As described here, the onReady callback is called when the new documents are ready, not when the old ones have been removed.

Your second workaround looks good. I have resisted the temptation to do something like this for several months now, hoping MDG would come back with, at least, an acknowledgment of the problem. Notice that this workaround is painful in many ways (for example it requires, on the client, that any query refers to the skip parameter).

Just for the sake of the discussion, here is what might be a (heavy) client-only hack:

    Collection.find({}).observeChanges({
      added: function (id, doc) { 
        Collection._collection.update(id, { $set: { requestId: requestId } }); 
      },
      changed: function (id, fields) {
        Collection._collection.update(id, { $set: { requestId: requestId } }); 
      }
    });

Regarding the requestId, a generic solution might be to use the new subscriptionId field of the subscription handle (see here).


#9

If we come to a conclusion here, we could post an answer to this SO question.


#10

the reason I had to resort to that solution is the fact the subscription ready is not working for me in an template.autorun scenario. Especially if I subscribe to more than one collection.

As for the second solution, the client knows the skip parameter, because it uses it to subscribe, so it is easy.

Where are you getting the subscription id from?


#11

The client sure knows. But you have to make it a global variable so that every module knows it. And then you have to change all your queries to depend on that global variable, which makes them non-generic. A useful hack, but a hack.

The doc says you get it from the client-side subscription handle (Meteor.subscribe(...).subscriptionId, not tested). Don’t know if there is a way server-side.


#12

doesn’t have to be global, can be in the template. Like in my original example, except that I had a typo: instead of var skip it should be self.skip

Template.gigaCollection.onCreated(function(){
 var self = this;
 self.skip = new ReactiveVar(0);
 var limit = 12;
 self.autorun(function(){
  self.subscribe('gigaCollection', {sort:{name:1},skip: self.skip.get(), limit: self.limit});
 })
});

Unsubscribe - subscribe issue with large users collection
#13

An even heavier client-side solution:


#14

I created a quick example on MeteorPad using the subscriptionId