Add subscriptionId to documents during subscription

core
1

When changing a subscription to a collection, there will be a window of time on the clients where minimongo will have results from both subscriptions. Unfortunately the subscription will be marked ready on the client as soon as the new subscription is ready, and not after the old subscription has been cleaned out. When iterating over a curser from the subscribed collection. Unfortunately the documents in the collection are oblivious to which subscription request they belong.

Fortunately DDP tracks the subscriptionId to manage publications on the server, and even exposes that id in the client subscription.

Wouldn’t it make sense to add the subscriptionId to the documents added in ddp/livedata_server.js? Here is the added method of the Subscription object

  added: function (collectionName, id, fields) {
    var self = this;
    if (self._isDeactivated())
      return;
    id = self._idFilter.idStringify(id);
    Meteor._ensure(self._documents, collectionName)[id] = true;
    fields._subscriptionId = self.subscriptionId;
    self._session.added(self._subscriptionHandle, collectionName, id, fields);
  },

With this addition it would be possible to filter the values on the client belonging only to the subscription.

Unfortunately I have not been able to figure out if that additional synthetic value should be removed when writing a record back to mongo.

I posted a demo of this in this forum post

2

Hm, but then since when more than one subscription gets a copy of the same document, meteor arbitrarily decides which document to send down to the client, we still won’t be able to have two copies of the document each one belonging to a separate subscription. So all we end up with will be document with a subscription id that will not make much use.

Or am I missing something here?

3

nope, I think you see that correctly. However, currently there is no way to distinguish. With this solution you could optionally filter by subscription at least. Obviously my little suggested hack does not take the merge box into account. Maybe it should be an array document._subscriptionIDs?

4

BTW you could achieve the exact same thing by applying a transform within the publish function as in

Meteor.publish('myPublication', function(){
  return MyCollection.find({},{transform: function(doc) {
    doc.publication: 'myPublication';
    return doc;
  }})
});

But again, multiple publications will overwrite each other within the mergebox.

A more elegant solution comes from https://atmospherejs.com/percolate/find-from-publication and in fact I was reading its source when I saw your post :smile:

I’m looking in porting that codebase to work with https://atmospherejs.com/reywood/publish-composite

EDIT:
And then there is this http://braindump.io/meteor/2014/09/20/publishing-to-an-alternative-clientside-collection-in-meteor.html

5

In my particular example, it is the same publication on the same collection with different parameters to subscribe. Either way, there should be a mechanism that identifies the records which subscription they are from. Since every pub/sub has a uniq id I figured that would be the best approach.

I actually didn’t think of a transform and used observeChanges in my example

6

For reference here is the corresponding Meteor issue (with interesting comments from MDG, but no solution so far):

1 Like
7

I got an idea from the issue that @Steve shared. Have not yet tried it, but I think it will work.

MyCollection = new Mongo.Collection('mycollection');
MyCollection.insert({foo: 'bar'});

if (Meteor.isServer) {
  Meteor.publish('myPublication1', function(){
    return MyCollection.find({},{transform: function(doc) {
      doc[myPublication1]: true;
      return doc;
    }})
  }); 
  Meteor.publish('myPublication2', function(){
    return MyCollection.find({},{transform: function(doc) {
      doc[myPublication2]: true;
      return doc;
    }})
  }); 
}

if (Meteor.isClient) {
  Meteor.subscribe('myPublication1');
  Meteor.subscribe('myPublication2');
  // assume we are in a reactive computation
  console.log(MyCollection.findOne());
  // should be {foo: 'bar', myPublication1: true, myPublication2: true}
}

Now if this works, and assuming that Meteor merges top level fields also within the transform, it means we’d know exactly where the data is coming from.

This should be checked for reactivity as well, since transforms have a problem with that as far as I remember.

8

I can’t get a transform in a publish to work at all …

9

That’s odd… Are you using a package that perhaps monkey patches Mongo.Collection?

10

Absolute vanilla, with just bootstrap and flow-router 2.x

 meteor list
insecure 1.0.3 Allow all database writes by default
kadira:blaze-layout 2.0.0 Layout Manager for Blaze (works well with FlowRouter)
kadira:flow-router 2.1.0* Carefully Designed Client Side Router for Meteor
meteor-platform 1.2.2 Include a standard set of Meteor packages in your app
mrt:bootstrap-3 0.3.8 Provides bootstrap 3.
natestrauser:animate-css 3.2.6 Animate.css packaged for meteor
 cat .meteor/release
METEOR@1.1.0.3

11

Hm, what exactly do you mean by not getting transforms work at all?

12

Ok, now I got it.

Take a look at http://meteorpad.com/pad/bykyfCtAu5mFxuurk/Publish%20with%20transform

The thing is, I’ve never used stock Meteor.publish(), instead I’ve always been using Meteor.publishComposite() from reywood:publish-composite and since it returns normal cursors, my transforms have been working as I expected them to.

But, evidentally, transforms do not work with find within publish or they do but I don’t know how.

http://docs.meteor.com/#/full/mongo_collection states:

transform Function

An optional transformation function. Documents will be passed through this function before being returned from fetch or findOne, and before being passed to callbacks of observe, map, forEach, allow, and deny. Transforms are not applied for the callbacks of observeChanges or to cursors returned from publish functions.

while http://docs.meteor.com/#/full/find states:

transform Function

Overrides transform on the Collection for this cursor. Pass null to disable transformation.

So I really don’t know what to make of this documentation now.

Perhaps the canonical way to do what I did here would to use observe but I really don’t like the verbose syntax there. So I guess as long as publishComposite works for me, I guess I’ll be leaning on this pattern.

Unless you or someone points out a flaw in my logic here, or perhaps a performance concern?

13

The reason this doesn’t work with observe changes is because the database stops being the authority on what the delta is, and observe changes only responds with the delta. Publications use observe changes instead of observe for performance reasons.

14

@lassombra yep, I get that. It is clear per the documentation. My current concern is if there are any side effects of using transforms within the publishComposite callback.

15

The only real side effect would be that it would have to use observe instead of observe changes which has a performance overhead in that it can’t calculate just the delta, it has to send the entire object over the pipe with every update.

16

I can’t find anywhere in the documentation that you can not have a transform of a Collection.find inside of publish. I can observeChanges inside of publish, which is how I initially implemented adding the subscriptionId.

The following works as expected if useTransform = false

Meteor.publish('collection', function(query, options) {
  query = query || {};
  options = options || {};
  var self = this;
  var index = 1;
  if(useTransform)
  {
    options['transform'] = function(doc) {
      console.log('publish transform',self._subscriptionId);
      doc._subscriptionId = self._subscriptionId;
      doc.index = index++;
      return doc;
    };
    // console.log(self._subscriptionId, options);
    return Collection.find( query, options );
  }
  else
  {
    var handle = Collection.find( query, options ).observeChanges({
      added: function (id, fields) {
        fields._subscriptionId = self._subscriptionId;
        fields.index = index++;
        self.added('collection', id, fields);
      }, // Use either added() OR(!) addedBefore()
      removed: function (id) {
        self.removed('collection', id);
      }
    });
    self.ready();
    self.onStop(function () {
      handle.stop();
    });
  }
});
17

As mentioned at http://docs.meteor.com/#/full/mongo_collection

transform Function
An optional transformation function. Documents will be passed through this function before being returned from fetch or findOne, and before being passed to callbacks of observe, map, forEach, allow, and deny. Transforms are not applied for the callbacks of observeChanges or to cursors returned from publish functions.

Using Observe instead of ObserveChanges is one way you can do that but you then

  • Increase processor overhead because you are processing an entire object on each change
  • Increase delays as Meteor makes additional requests to Mongo for the entire object (instead of just the changes which it gets from oplog)
  • Increase the amount of data going over the ddp connection to the client as it is publishing whole objects instead of just updates.
  • Increase the processor overhead on the client as the mergebox tries to handle the larger updates and triggers massive reruns of autorun scopes all over the place.

It’s up to you and your environment to decide if this is worth the price, but that is why by default publications don’t respect transforms.

18

I have read that. The question was why transform defined with a find is not working in publish on the server.

19

Maybe because publish uses observeChanges under the hood when it is fed a cursor.

20

@serkandurusoy Good point.

So the only way to add the subscriptionId is to publish a cursor with observeChanges yourself.

I still think it would be a good feature for core to optionally enable a similar feature.

3 Likes