Non reactive publications in meteor

Sorry to revive, so in your tests you found that reactive: false is essentially like using a Meteor method then? Does it eliminate all the server overhead with observeChanges and Tracker.autorun etc. etc.?

1 Like

A method would only be able to return the result in an array. With reactive: false you would be able to work with the collection on the client.

Using {reactive: false} on the client has nothing to do with the server overhead. It simply means that it will not create a Tracker dependency and the cursor will not be invalidated when new data is available (when used in a reactive context, such as a Blaze template or a Tracker.autorun.

The method suggested by @dburles should work. It sends the required messages to the client, the same way a normal, ā€œreactiveā€ publication would, but will not keep processing the cursor after the initial fetch.

1 Like

A couple questions - forgive my inexperience. On Kadira, Iā€™m having lots of delays on observeChanges on my subscriptions and Iā€™m wanting to strip out reactivity on a lot of things. Whatā€™s the best approach for eliminating as much observeChanges as possible? I want my actual reactive subscriptions to only push documents and fields that are required to be reactive for my app to function. Everything else can be static and/or updated on refresh.

A Meteor Method probably has the least overall overhead to the server. If instead I use {reactive: false} doesnā€™t this eliminate the observeChanges overhead on the server/DB which does in fact improve server overhead? Or on the server it still observes changes thus adding overhead, but not on the client?

Lastly, if I used the @dburles method, should I also throw in a this.stop() at the end just to close down the subscription?

1 Like

This has no influence on the server. It is a MiniMongo cursor. The server is not aware of those.

This solution is not a complete one. Adding onStop is likely the first enhancement you can make, although it adds a bit of memory and processing overhead (of course, this is needed in order to clean up after the publication). Other options are publishing to a dedicated client-only collection and tossing it away afterwards (this can also be done when using a method).

@alon, is it ok to use the dburles method as-is even though it is ā€œnot a complete oneā€? i.e. is it simply somewhat inefficient, or is it going to break randomly because of race conditions, etc?

That really depends on your use case.

It will send some documents to the client and thatā€™s it. It will not clean up after itself when you unsubscribe, so in some use cases, you will have unwanted documents in your client-side collection. Using a method to non-reactively retrieve data may be more efficient in some cases.

If you can live with that, I see no real issue with using it.

1 Like

Sorry to revive this again, but I went down the long road of converting a lot of pub-sub to Meteor Methods to avoid the overhead of pub-sub only to realize that if you have 1000 users all requesting the same data via a Meteor Method (e.g. a chatroomā€™s Chats for example) it will hit Mongo 1000 times for the exact same data which will (and did) create a bottleneck.

I found itā€™s better to use pub-sub if you have heavy data/observer reuse. That way itā€™s way lighter on Mongo by sending data to the clients thatā€™s already on the server versus re-fetching it from Mongo every Method call. If you donā€™t need reactivity, the above ā€œstatic publicationā€ patterns work well.

Thereā€™s also cult-of-coders:grapher than can cache Meteor Method-like queries but this seems very similar to a nonreactive static publication.

Iā€™m now understanding everything much more after a few years of being into Meteor.

4 Likes

Couldnā€™t you memoize your Meteor method function and invalidate the cache each second?

Then youā€™re essentially debouncing queries to the database and serving data that may be up to a second out of date to everyone else

1 Like

Will you share a nonreactive static publication example with us?

@coagmano Yes I guess you could memoize. I actually have never heard of that until right now. Iā€™m not the savviest programmer in the world. This article on using lodash memoize in Meteor is interesting. I wonder if cult-of-coders:grapher uses this behind the scenes (checking itā€™s package.js on Github it has a dependency on lodash). Very cool though. Wouldnā€™t this basically do what a static nonreactive publication would do if the query is the same? For example, 1000 users getting the same chat room history. Publications ā€œdebounceā€ queries to the database for the same query. Iā€™ll have to do some cloud tests to see which pattern is more performant under user load. The Meteor Method memoize route probably uses less memory than the pub-sub overhead. Maybe not if itā€™s a static nonreactive publication and isnā€™t observing.

@aadams Read this thread, the code is above. Youā€™re basically doing a manual publication accessing the this functions available in it to control how documents get added to the cursor. So calling this.added(...) to add each document into the cursor, but then not calling a this.changed(...) or a this.removed(...) which strips out the reactivity because it only adds documents to the cursor. Then call this.ready() and this.stop() to stop new documents from being added. That would be a static nonreactive publication that acts like a cache for the same query. Then it probably doesnā€™t hurt to add reactive: false to any find(..) calls on the client after subscribing.

The point of doing this ā€œnonreactive staticā€ publication though, for me at least, is to get a ā€œcacheā€ functionality off-the-shelf with core Meteor.

1 Like

I was implementing the pattern described in this thread and I wanted to mention an issue with the code above: I said that publications can act like a cache and debounce subscribe calls off Mongo to the server because subscribing with the same query will use the data already on the server and not hit mongo again.

I told @aadams to look at the code above. But Iā€™m thinking the code above is wrong if you want to achieve this because it has a fetch() in it, which will hit Mongo every time (I think - unless Meteor also handles fetch() calls with the same query within publications). To have observer reuse you have to use an observer.

So instead of fetching for the documents like this:

Meteor.publish('example', function() {
  const docs = Foo.find().fetch();
  docs.forEach(doc => this.added('foo', doc._id, doc));
  this.ready();
  this.stop();
});

It needs to use an observer:

Meteor.publish('example', function() {
  var self = this;
  var fooHandle = Foo.find().observe({
    added: function (addedFoo) {
      self.added('foo', addedFoo._id, addedFoo);
    }
  });
  self.ready();
  //self.stop();  //  EDIT: Do not do this, will not deliver any data to client

  // Instead do this
  self.onStop(function() {
    fooHandle.stop();
  });
});

The one thing Iā€™m wondering now is: I want to call self.stop() to stop any reactivity and overhead (because I just need a cached Meteor Method functionality). But Iā€™m curious if self.stop() kills the observer reuse and subsequent calls from other clients to this publication will re-query Mongo?

Per the Meteor doc it says self.stop() only stops the clientā€™s subscription, but Iā€™m wondering if itā€™s cleaned up on the server too. So having 1000 simultaneous calls to this publication would then hit Mongo 1000 times?

If it did, I could always just not call this.stop() in the publication and either benefit from new documents that get added, or use reactive: false on the client to stop any added document updates if I truly donā€™t want them and/or immediately call stop() on the client to kill any reactivity.

EDIT: After some experimentation, I answered my own question about calling this.stop(). The answer is you cannot call this.stop() in the publication like I did above. If you do this, the data never reaches the client. Instead, you can listen for the subscriptionHandle.stop() and then call this.stop() (I modified the code above).

However, even if you try to stop the subscription on the client immediately after calling subscribe() by calling stop() on the client subscription handle like this:

subscriptionHandle.stop() or instance.subscriptions[x].stop()

ā€¦or by allowing the template to be destroyed, the data is also removed from the client. So you really canā€™t call stop() how I was thinking (and how itā€™s mentioned above in this thread). Any call to stop() kills the data. So, you either deal with reactivity of new documents being added or call reactive: false on the client Foo.find().

EDIT2: If you were dead-set on calling subscriptionHandle.stop() to somehow end reactivity in a pub-sub, you could probably subscribe to the data on the client, then in the callback of the subscribe do a var foos = Foos.fetch() on the data and save it locally to use, then immediately call subscriptionHandle.stop(). No reactivity and you have the data. This would effectively be very much like a Meteor Method.

The problem is Iā€™m not totally sure what stop() does to a publicationā€™s observer on the server. If it kills that observer on the server right away, then it might be a bad idea to stop() like this as it may never leave you with a cached publication observer.

Imagine 1000+ users all subscribing, fetching, then stopping at the same time. If Meteor is somehow smart enough to always let one observer linger for a while on the server, then they would all hit the cache. But if theyā€™re all calling stop() as fast as they can and stop() kills the publicationā€™s observer then there would likely never be an existing cached observer in existence (or at least not all the time). So you would have a lot of Mongo hits because no existing observer is in existence. Probably best to not stop() until your app does it naturally (e.g. user navigates away from a template and its subscription is torn down).

Sorry this got so meta. This is worth having an MDG dev chime in.

2 Likes