This is really a cool improvement on top of pubsub and methods. And as mentionned, it can be the simplest step to take to have a better scalability on your app before redis-oplog for example, which is easier in term of deployment ( no other server to setup ).
I have bookmark this to try it in one of my project.
Cheers
Looks amazing, bravo!
What are the minimum required (or recommended) Meteor & Mongo versions supported?
This is indeed a very interesting idea. It’d be great if we can keep the simplicity of pub-sub-lite while using grapher’s powerful querying layer under the hood.
I’ll give this idea a try during the weekend. Thanks for your suggestion!
Thank you! It will be great to see how well the package works with your app. Can’t wait to hear your feedback!
Yeah that was my main goal when I started developing the package: Providing a way to get rid of unnecessary pub/sub in an existing codebase with minimal effort. Your feedback on the package will be very welcomed!
Thanks Matt! The package will work with most Meteor versions, as it simulates the signature and behaviours of Meteor pub/sub which have long been stable. On the MongoDB side, if you use the mutation update messages emitting feature, you will need at least MongoDB 3.6.0 and a replica set, in order to support Change Streams (which the package relies on for getting updates from MongoDB). More information can be found here.
I find this library very interesting as I’m a big fan of Meteor methods and its simplicity.
But please help me to understand this better, I’ve not looked into the code yet.
Under the hood, are you completely replacing Meteor’s pub/sub mechanism with observing the methods for updates? What happens if someone update the record directory in the DB without using a Meteor method, will it emit the update events or is still piggybacking on Meteor pub/sub? I’m just trying to understand how this is working better.
As I have described here, the package doesn’t aim to obsolete pub/sub in Meteor. It just helps you quickly convert existing pubs/subs (that you’ve identified as unnecessary) into Methods with minimal effort. It also provides an enhanced version of Methods that is more convenient to use (with added features such as caching, Minimongo merging, and mutation update messages emitting).
Enhanced Methods are just traditional Meteor Methods by nature. In order to keep track of changes to documents during a server-side invocation, it makes use of MongoDB’s Change Streams. All detected changes will be sent to the client caller via DDP messages. So enhanced Methods can’t completely replace pub/sub because it cannot keep multiple clients in sync, but it can help the Method caller be aware of all server-only changes that traditionally can be retrieved only by pub/sub or by manual logic.
So I would envision that the package will be used in the following way:
Meteor.publishLite and Meteor.subscribeLite to replace existing unnecessary pub/sub.Meteor.methodsEnhanced and Meteor.callEnhanced/applyEnhanced for retrieving and mutating non-reactive data.Sorry, just to confirm my understanding, this change event is broadcasted to the caller client only (not other clients) and will automatically merge the changes to the caller client mini-mongo, is the correct?
Am I correct that this is a good mechanism for offline PWA? Methods caching the results through minimongo?
If I not mistaken, I think it caches the returned results but I will not client cache when offline and auto-sync when online.
MongoDB has a new solution for this called realm, I think it is worth exploring.
However, I do think thing there is opportunity to extend methods to support this mechanism, I think it would be great package.
Hi,
Would you mind elaborate a little on this, please ?
Only downside is minimongo. Which seems ti be flushed regularly, in order to lighten the client RAM I presume. Unfortunatly, grapher doesn’t propose to turn this behavior off, on a query basis.
Yes, that’s correct. The goal of this feature (mutation update messages emitting) is to address an issue you may face after getting rid of a particular pub/sub: When adding/updating/removing documents via a Method, the client-side caller no longer knows what changes have happened on the server. Traditionally that knowledge can only come from 1) having those changes sent via a pub/sub or 2) writing manual logic. Enhanced Methods automate this so your mutation Method will continue to work as if it was backed by a pub/sub (but instead of all clients, only the Method caller will receive these updates).
The package’s caching layer only caches Method result data, with the goal of avoiding unnecessary repeated calls. For offline PWAs we will need a more comprehensive solution. MongoDB Realm looks promising - it’s great to see that the MongoDB ecosystem is going in the right direction. I’ll think about use cases for PWAs further. Thanks for your suggestions!
I think @csiszi meant the following problems, given the context of using grapher in a non-reactive setting:
Imagine you have a todo app and the todo list lives in a <TodoList> component which uses a reactive grapher query the get the items. Every item has a switch to mark it as done. The switch uses a regular meteor method to update the completed field of that todoItem. Because of the reactive query, the change is sent down to <TodoList> without any additional logic (i.e. the name of the todo item is now crossed out).
If you change the reactive query to a static query, the <TodoList> component works at first, but when an item is clicked and the update method is called, the static query in <TodoList> doesn’t re-run so the change is not reflected in the UI. You have to manually get the updated todo item and merge it into the array retrieved using the static query or run the entire static query again.
To be honest grapher doesn’t have another feature which would be awesome: to use the data in the minimongo without running the query on the server. Let’s say you have another component which only shows the name fields of the todos. It’s a sibling of <TodoList> and if <TodoList> has a ready reactive query, there’s no need to ask the server for the names, we already have them in minimongo.
With grapher-react, one could jusy mark the request as non-reactive, with a boolean flag like so :
export default withQuery(
props => {
return getPostLists.clone();
},
{ reactive: true /*|| false*/ },
)(PostList);
This is very handy and doon’t need any refactoring clientside.
Do you mean that with your package, serverside changes are synced cliendside, without being fetched on a websocket ? Is it constant HTTP polling ?
Seems to be worse in term of performance, ain’t it ?
Sure, I’m really interested to hear how pub-sub-lite could heandle this differently.
Long polling ? 
Sorry I should have been more specific. I was mentioning about what will usually happen when you use pub/sub predominantly in your app and then decide to switch certain parts to using Methods: It will require a lot of refactoring, especially on client-side because of the difference in signature and behaviours between pub/sub and Method. The pub-sub-lite package solves this issue because the helpers it provides (Meteor.publishLite and Meteor.subscribeLite) simulate the signature and behaviours of their native counterparts (Meteor.publish and Meteor.subscribe), so your existing rendering logic can mostly remain intact. If your app uses grapher-react (as in the code snippet you provided) then there is no use for pub-sub-lite.
There is no polling involved. pub-sub-lite’s enhanced Methods keep track of changes made during a server-side Method invocation by using MongoDB Change Streams, and then send those changes in the form of DDP messages to the client-side Method caller, where the changes will be automatically merged into Minimongo. This will resolve the issue mentioned by @csiszi in his example scenario above.
Thanks for the in-depth explanation.
I’ll dig in, in order to anderstand how the method caller could be called back without listening to a DDP stream set before…?!
It works like this: When a DDP client connects to the Meteor server, that connection is represented by a Session instance. All sessions (representing all DDP clients) are stored in Meteor.server.sessions. Each session carries a unique id, and this id is also attached to the invocation context (the familiar this) of each Method call. So we can use this attached id to trace back to the client session who called the Method:
const allConnectedClientSessions = Meteor.server.sessions;
Meteor.methods({
myMethod() {
/*
Note: This code is for illustrative purposes only. When using
pub-sub-lite's enhanced Methods you just write your mutations
normally without the need to know any of these details.
*/
// `this` is the invocation context mentioned above
const clientSessionId = this.connection.id;
const clientSession = allConnectedClientSessions.get(clientSessionId);
// Each session carries a `send` method, allowing server to send
// messages to that particular client
clientSession.send({
msg: 'changed',
collection: 'books',
id: 'NzrGsj9ooJnQwbDfZ',
fields: { numberOfBooksSold: 99 },
});
}
});
The changed message above is structured according to the DDP specification. When receiving this standardised message, the client will automatically update Minimongo to reflect the change.
You can find the relevant code in pub-sub-lite here.
First, this sounds like a great performance boost for Meteor apps that scale. I’ve read through this thread and feel like it has terrific promise. This quote is the part of things where I’m unsure what the change would be using this vs true pub/sub.
Given a Meteor app that has a social component where many parties are connected clients seeing the same thing (like a project management app), I’m imagining a scenario of 10 connected clients where one of them triggers a change that propagates through pub/sub lite. If the attached id is traced back to the client session who called the Method, does that mean that the one client that triggered will see the change, while the other 9 that didn’t call the Method won’t see it?
Sorry if I’m missing things but just want to understand how this would work in an app with many connected clients all tuned into (essentially) one data source.
I think npvn answered this question here:
Only the caller mini-mongo is updated, not the other 9 clients.
However, perhaps the solution could be extended to update the other nine, then it’ll be another version of redis-oplog.