☄️ Introducing pub-sub-lite: Lighter (Method-based) pub/sub for Meteor

announce
1

Hi everyone!

At MaestroQA we have been doing an architectural refactoring to improve the performance of our app. One of the goals is to identify the parts of the application that don’t really need reactivity, and replace existing pub/sub code in those parts with Method calls for fetching data.

However, using Method calls for fetching data has certain disadvantages:

  • Switching an existing codebase from using pub/sub to using Methods usually requires a lot of refactoring, especially on the client-side.

  • Data retrieved from Methods is not merged into Minimongo, which means other parts of the front-end cannot make use of this data.

  • Each Method call will trigger a separate request from the front-end, resulting in a separate invocation on the back-end. This means if a particular Method is called repeatedly with the same arguments, we will end up wasting resources processing and sending the same data multiple times.

  • Without pub/sub, the changes made to documents during a mutation Method are not sent to client. It will be impossible for the client to always reflect these changes accurately, because there can be server-only logic that cannot be simulated with optimistic updates on the client-side. We can refactor the Method to manually return the changed documents and merge them to Minimongo, but that requires a lot of custom code.

To solve these problems, we created pub-sub-lite, a Meteor package that provides:

  • Easy-to-use helpers (Meteor.publishLite and Meteor.subscribeLite) that can be used in place of Meteor.publish and Meteor.subscribe. No other change is necessary. The original pub/sub will be converted to a Method call and the result data will be merged to Minimongo automatically. Meteor.subscribeLite also provides a handle that reactively returns true once data has arrived, which means existing front-end code can be kept intact!

  • Enhanced Methods (Meteor.methodsEnhanced, Meteor.callEnhanced and Meteor.applyEnhanced):

    • Automatically merge result data into Minimongo.
    • Automatically send server-side document changes made during a Method invocation to the client-side caller and merge these changes to Minimongo, without using any pub/sub.

  • Caching layer:

    • Automatically deduplicate unnecessary repeated Method calls, with customizable cache duration that can be set globally or individually per Method.
    • Result data caching: Method result data is cached and can be synced with Minimongo (when data contains documents) before being returned when there is a cache hit.

Use cases
pub-sub-lite will be useful when:

  • You have used pub/sub predominantly in your codebase and now want to switch certain parts to using Methods, without doing a lot of refactoring.
  • You want to leverage the benefits of enhanced Methods (caching, Minimongo merging, mutation updates emitting).

More information about the package can be found at https://github.com/adtribute/pub-sub-lite.

We hope to receive your feedback to further improve the package!

54 Likes
2

This is exactly what I’m missing in grapher. Grapher is super-cool to fetch data from related (linked) collections but changing from pubsub to a method causes the problems you mentioned. If only these two would work together…

2 Likes
3

Congrats! This is really cool. Especially the drop-in replacement and caching.

We’ve been avoiding pub-sub whenever possible - it’s the cause of so many issues - and replacing with methods (imho the underestimated killer feature of Meteor). But that introduces new problems you mention here and now you’ve created a solution for that.

Awesome! Will definitely give it a try!

3 Likes
4

This is really a cool improvement on top of pubsub and methods. And as mentionned, it can be the simplest step to take to have a better scalability on your app before redis-oplog for example, which is easier in term of deployment ( no other server to setup ).

I have bookmark this to try it in one of my project.

Cheers

1 Like
5

Looks amazing, bravo!

What are the minimum required (or recommended) Meteor & Mongo versions supported?

1 Like
6

This is indeed a very interesting idea. It’d be great if we can keep the simplicity of pub-sub-lite while using grapher’s powerful querying layer under the hood.

I’ll give this idea a try during the weekend. Thanks for your suggestion!

7

Thank you! It will be great to see how well the package works with your app. Can’t wait to hear your feedback!

Yeah that was my main goal when I started developing the package: Providing a way to get rid of unnecessary pub/sub in an existing codebase with minimal effort. Your feedback on the package will be very welcomed!

Thanks Matt! The package will work with most Meteor versions, as it simulates the signature and behaviours of Meteor pub/sub which have long been stable. On the MongoDB side, if you use the mutation update messages emitting feature, you will need at least MongoDB 3.6.0 and a replica set, in order to support Change Streams (which the package relies on for getting updates from MongoDB). More information can be found here.

2 Likes
8

I find this library very interesting as I’m a big fan of Meteor methods and its simplicity.

But please help me to understand this better, I’ve not looked into the code yet.

Under the hood, are you completely replacing Meteor’s pub/sub mechanism with observing the methods for updates? What happens if someone update the record directory in the DB without using a Meteor method, will it emit the update events or is still piggybacking on Meteor pub/sub? I’m just trying to understand how this is working better.

3 Likes
9

As I have described here, the package doesn’t aim to obsolete pub/sub in Meteor. It just helps you quickly convert existing pubs/subs (that you’ve identified as unnecessary) into Methods with minimal effort. It also provides an enhanced version of Methods that is more convenient to use (with added features such as caching, Minimongo merging, and mutation update messages emitting).

Enhanced Methods are just traditional Meteor Methods by nature. In order to keep track of changes to documents during a server-side invocation, it makes use of MongoDB’s Change Streams. All detected changes will be sent to the client caller via DDP messages. So enhanced Methods can’t completely replace pub/sub because it cannot keep multiple clients in sync, but it can help the Method caller be aware of all server-only changes that traditionally can be retrieved only by pub/sub or by manual logic.

So I would envision that the package will be used in the following way:

  • Meteor.publishLite and Meteor.subscribeLite to replace existing unnecessary pub/sub.
  • Meteor.methodsEnhanced and Meteor.callEnhanced/applyEnhanced for retrieving and mutating non-reactive data.
  • Traditional pub/sub for reactive data.
3 Likes
10

Got it @npvn, thanks a lot for this awesome and well-thought package, can’t wait to try it.

1 Like
11

Sorry, just to confirm my understanding, this change event is broadcasted to the caller client only (not other clients) and will automatically merge the changes to the caller client mini-mongo, is the correct?

12

Am I correct that this is a good mechanism for offline PWA? Methods caching the results through minimongo?

2 Likes
13

If I not mistaken, I think it caches the returned results but I will not client cache when offline and auto-sync when online.

MongoDB has a new solution for this called realm, I think it is worth exploring.

However, I do think thing there is opportunity to extend methods to support this mechanism, I think it would be great package.

1 Like
14

Hi,

Would you mind elaborate a little on this, please ?

  • Gapher has caching + denormalization bundled
  • grapher-react has a flag to turn reactivity on and off from the query

Only downside is minimongo. Which seems ti be flushed regularly, in order to lighten the client RAM I presume. Unfortunatly, grapher doesn’t propose to turn this behavior off, on a query basis.

15

Yes, that’s correct. The goal of this feature (mutation update messages emitting) is to address an issue you may face after getting rid of a particular pub/sub: When adding/updating/removing documents via a Method, the client-side caller no longer knows what changes have happened on the server. Traditionally that knowledge can only come from 1) having those changes sent via a pub/sub or 2) writing manual logic. Enhanced Methods automate this so your mutation Method will continue to work as if it was backed by a pub/sub (but instead of all clients, only the Method caller will receive these updates).

The package’s caching layer only caches Method result data, with the goal of avoiding unnecessary repeated calls. For offline PWAs we will need a more comprehensive solution. MongoDB Realm looks promising - it’s great to see that the MongoDB ecosystem is going in the right direction. I’ll think about use cases for PWAs further. Thanks for your suggestions!

I think @csiszi meant the following problems, given the context of using grapher in a non-reactive setting:

  • The need to have a lot of refactoring, especially on the front-end.
  • The lack of Minimongo merging, which means the data can’t be used elsewhere.
  • The lack of knowledge about what have actually changed on the server-side (without pub/sub or manually sending the changes with custom logic).
2 Likes
16

Imagine you have a todo app and the todo list lives in a <TodoList> component which uses a reactive grapher query the get the items. Every item has a switch to mark it as done. The switch uses a regular meteor method to update the completed field of that todoItem. Because of the reactive query, the change is sent down to <TodoList> without any additional logic (i.e. the name of the todo item is now crossed out).

If you change the reactive query to a static query, the <TodoList> component works at first, but when an item is clicked and the update method is called, the static query in <TodoList> doesn’t re-run so the change is not reflected in the UI. You have to manually get the updated todo item and merge it into the array retrieved using the static query or run the entire static query again.

To be honest grapher doesn’t have another feature which would be awesome: to use the data in the minimongo without running the query on the server. Let’s say you have another component which only shows the name fields of the todos. It’s a sibling of <TodoList> and if <TodoList> has a ready reactive query, there’s no need to ask the server for the names, we already have them in minimongo.

2 Likes
17

With grapher-react, one could jusy mark the request as non-reactive, with a boolean flag like so :

export default withQuery(
    props => {
        return getPostLists.clone();
    },
    { reactive: true /*|| false*/ },
)(PostList);

This is very handy and doon’t need any refactoring clientside.

Do you mean that with your package, serverside changes are synced cliendside, without being fetched on a websocket ? Is it constant HTTP polling ?
Seems to be worse in term of performance, ain’t it ?

18

Sure, I’m really interested to hear how pub-sub-lite could heandle this differently.
Long polling ? :face_with_raised_eyebrow:

19

Sorry I should have been more specific. I was mentioning about what will usually happen when you use pub/sub predominantly in your app and then decide to switch certain parts to using Methods: It will require a lot of refactoring, especially on client-side because of the difference in signature and behaviours between pub/sub and Method. The pub-sub-lite package solves this issue because the helpers it provides (Meteor.publishLite and Meteor.subscribeLite) simulate the signature and behaviours of their native counterparts (Meteor.publish and Meteor.subscribe), so your existing rendering logic can mostly remain intact. If your app uses grapher-react (as in the code snippet you provided) then there is no use for pub-sub-lite.

There is no polling involved. pub-sub-lite’s enhanced Methods keep track of changes made during a server-side Method invocation by using MongoDB Change Streams, and then send those changes in the form of DDP messages to the client-side Method caller, where the changes will be automatically merged into Minimongo. This will resolve the issue mentioned by @csiszi in his example scenario above.

20

Thanks for the in-depth explanation.
I’ll dig in, in order to anderstand how the method caller could be called back without listening to a DDP stream set before…?!

1 Like