New package - jam:pub-sub - Method-based and Change Streams-based pub/sub, and subscription caching

jam:pub-sub brings three key features to Meteor apps:

  1. Method-based publish / subscribe with reactivity for the user when they make database writes
  2. Change Streams-based publish / subscribe
  3. Subscription caching

Method-based publish / subscribe
Meteor’s traditional publish / subscribe is truly wonderful. However, there is a cost to pushing updates reactively to all connected clients – it’s resource intensive and will eventually place limits on your ability to scale your app.

One way to reduce the need for the traditional publish / subscribe is to fetch the data via a Meteor Method but there’s a big problem here: the data won’t be automatically merged into Minimongo and you completely lose Meteor’s magical reactivity. Minimongo is great to work with and makes things easy as the source of truth on the client. Without it, you’ll need to create your own stores on the client and essentially duplicate Minimongo.

With jam:pub-sub, you use Meteor.publish.once and the same Meteor.subscribe to have the data fetched via a Meteor Method and merged automatically in Minimongo so you can work with it as you’re accustomed to. It also automatically preserves reactivity for the user when they make database writes. Note that these writes will not be broadcast in realtime to all connected clients by design but in many cases you might find that you don’t need that feature of Meteor’s traditional publish / subscribe.

Here’s a quick look on how you use it:

Create a publication using Meteor.publish.once

// server
Meteor.publish.once('notes.all', function() {
  return Notes.find(); // expects a cursor or an array of cursors just like Meteor.publish
});

Subscribe like you normally would on the client:

// client
Meteor.subscribe('notes.all')

That’s it. By using Meteor.publish.once , it will fetch the data initally and automatically merge it into Minimongo. Any database writes to the Notes collection will be sent reactively to the user that made the write.

Change Streams-based publish / subscribe
With jam:pub-sub and MongoDB Change Streams, you can preserve Meteor’s magical reactivity for all clients while opting out of the traditional publish / subscribe and its use of the oplog. Use Meteor.publish.stream instead of using Meteor.publish and subscribe using the same Meteor.subscribe on the client.

Here’s a quick look on how you use it:

Create a publication using Meteor.publish.stream

// server
Meteor.publish.stream('notes.all', function() {
  return Notes.find(); // expects a cursor or an array of cursors just like Meteor.publish
});

Subscribe like you normally would on the client:

// client
Meteor.subscribe('notes.all')

That’s it. By using Meteor.publish.stream , any database writes to the Notes collection will be sent reactively to all connected clients just as with Meteor.publish but you’ll be using MongoDB Change Streams instead of the oplog.

Subscription caching
Normally, when a user moves between routes or components, the subscriptions will be stopped. When a user is navigating back and forth in your app, each time will result in a re-subscribe which means more spinners, a slower experience, and is generally a waste.

By caching your subscriptions, you can create a better user experience for your users. Since the subscription itself is being cached, the data in Minimongo will be updated in the background until the cacheDuration expires for that subscription at which point it will be stopped and the data will be removed from Minimongo as expected.

If you’re familiar with ccorcos:subs-cache or other subscription cache managers, this would act as a replacement.

It’s not enabled by default to align with the current Meteor.subscribe behavior. You can turn it on globally for all subscriptions:

import { PubSub } from 'meteor/jam:pub-sub';

PubSub.configure({
  cache: true // defaults to false
});

Or you can turn it on for individual subscriptions when you subscribe:

Meteor.subscribe('notes.all', { cache: true }) // overrides the global PubSub config

You can also customize the cacheDuration globally and/or when you subscribe. The default is 1 minute from when the component was destroyed and the subscription was initially set to stop.

If you end up taking it for a spin, let me know! If you have feedback, feel free to post below or send me a DM.

17 Likes

Hey Jam,

This is great, nice work :clap: . I guess you got a lot of inspiration from pub-sub-lite? We had tried to use that package but ran into some issues with the cache and never managed to fix them.

So we’ll definitely give this a try and see how we fare. I think this really is the missing tool we need in Meteor as the convenience of minimongo is great, but currently the price for using it is reactivity.

You mention it being reactive for the current user. Is there any more detail on how this is handled?

We currently use ccorcos:subs-cache … so we can expect the same behaviour (for the caching element) if we switch ?

2 Likes

Thanks Mark!

Yep! This package was inspired by pub-sub-lite but takes a different approach on implementation.

Basically, this package gets the current MethodInvocation when a write is called, grabs the session info, and then sends the DDP message(s) when writes are successful.

The relevant code is here. There’s a little bit of message filtering too in ddp.js.

Yes. One difference is the default cacheDuration, aka expireAfter in ccorcos:subs-cache. I set the global default to 60 seconds rather than 5 min. It’s also set in seconds not minutes.

I also debated to globally defaulting subscription caching to true. For now, I took a more conservative approach and defaulted it to off. Curious if you have thoughts on either of those global defaults.

Amazing work as usual @jam. Actually, I’m interested in something else entirely and would to use this effort to spring board it.

I believe GitHub - ccorcos/meteor-subs-cache: A Meteor package for caching subscriptions. should be added to the core, it offers a very simple yet intrinsic fix to a critical flaw that the OG developers overlooked.

This is almost identical to unblock, it started off as a package yet it was added to the core later on.

ccorcos:subs-cache as a code is very little and given you came up with replacement for it, I think you’re very capable of coming up with PR to merge into the core.

This was actually on my to-do list either way.

cc @nachocodoner cuz you’re a core dude so if it’s a good idea, you can greenlight it

cc @zodern cuz monti and stuff

2 Likes

Thanks Harry! Honestly, I would love to see this entire package become part of Meteor core. To that end, I made it as native feeling as possible with the API. The Method-based pub/sub may even be more important than subscription caching.

I think Mark said it best:

There are certainly ways to integrate this even more seamlessly into Meteor core. I’m sure the Meteor core developers would have some great ideas too.

Appreciate the compliment. At this time, I don’t think I’ll have the bandwidth for a PR to Meteor core, but I may have some windows open up down the line. In the meantime, if the Meteor core developers like the concept, I’m sure it would be easy for them to pick up.

2 Likes

@denyhs @grubba @nachocodoner @zodern

1 Like

This looks fantastic! Should it be compatible as-is with redis-oplog? (I gave it a spin in our app which but am not seeing writes propagate reactively back to the user that initiate the write – thinking perhaps it is because we’re also using redis-oplog, but could also be my error!)

Thanks! I didn’t test it with redis-oplog but I think it should work just fine. Can you share a code sample of what you have or post a small repro somewhere?

@brianlukoff Update: I did a quick test with redis-oplog and it seems to be all good. If you can help with a repro, maybe we can track down what’s going on. One thing to note is that jam:pub-sub expects that you’ll be using the *Async collection methods in your Meteor Methods.

Hi Jam, looks awesome (as usual)!

Regarding reactivity, I just wanted to clarify:

Any database writes to the Notes collection will be sent reactively to the user that made the write.

So basically we can think of this as pub-sub without needing the oplog, e.g. for each DB write, we look at our session’s current user, check what they’re subscribed to, and replicate any writes on the server down to the client?

Also, just to clarify from the readme:

Important: when naming your publications be sure to include the collection name(s) in it. This is generally common practice and this package relies on that convention.

I’m not 100% sure I’m following that convention :sweat_smile: But how specific are the rules about this? E.g. I’ll do something like orgorappname/domain/subdomain/action/qualifier for method and publication names. Should I be using periods instead of slashes, and does the position of the collection name matter?

I also have some subscriptions which need to do some filtering based on other conditions, e.g. ownership of a document, which requires querying one collection (e.g. the current user or team), then querying another (e.g. documents they would own).

If a user made a “write” that, for example, “gave away” a document they owned, would the publication still propagate that write down to the client, giving them the document they “gave away”?

(Obviously there’s “defence in depth” techniques here that would need to be considered, e.g. just because they can subscribe to a document they no longer own, doesn’t mean methods mutating that document shouldn’t check for ownership… but just trying to figure out how different the behaviour is)

1 Like

Thanks!

For the user making writes, yes. Here’s a simple example: let’s say you have two separate users, Alice and Bob. They both are subscribed to the notes.all publication that you set up with:

Meteor.publish.once('notes.all', function() {
  return Notes.find();
});

Now let’s say Bob creates a new note via a Meteor Method. When the note successfully inserts, Bob will see it reactively but if Alice was logged in at the same time she wouldn’t see it until she re-subscribed, e.g. she refreshed the page or went somewhere else in the app and came back.

As long as you have the name of the collection somewhere in the name of the publication, you’re good to go. And when I say name of the collection I mean the name you assign it with:

new Mongo.Collection(// the name you put in here //)

I’m not sure I completely follow. For the writes, it uses whatever logic is contained within your Meteor Methods. If the write is successful, then the result is sent back to the client that called the Method.

1 Like

Thanks! I will dig in and try to figure out what is going wrong. Do you mean that the function passed to Meteor.publish.once should be marked as an async function? Or is there another place we need to be using the *Async functions (the publication itself is just calling Collection.find)?

I mean the Collection method inside your Meteor.methods, e.g. if you have a Notes Collection

Meteor.methods({
  'insertNote': async function(…) {
    return Notes.insertAsync(…)
  }
});

Not the publication. Meteor.publish.once expects you to return a cursor, e.g. Notes.find(), or an array of cursors just like Meteor.publish.

Got it – thanks! That makes sense. I’m still having trouble getting the updates to reactively pass back to the same user, but I’ll dig in and see if I can find a repro.

(One thing I did notice is that passing an _id as a selector (i.e., as a shortcut for { _id: "..." }) doesn’t seem to work – it looks like mongo.js is expecting the selector to be an object and not an _id.)

Thanks. I just published a fix in v0.1.1 for the shorthand _id.

2 Likes

That is really cool, I love seeing packages that work with the meteor pub/sub!

I certainly will give it a try on the package.

2 Likes

Congrats on your work with jam:pub-sub. It is really nice see these features implemented in a package.

I agree that most of the time you don’t need reactivity on your app, and using methods instead of publications is convenient for performance reasons. In fact, in the projects I have participated in we have had the exact same concept implemented as a helper, fetching via methods and conciliate the data on minimongo in a performant way (pause and resume observers as you do). I have adopted this as well on my projects, though I just distinguished between Queries and Mutators, being Queries able to be reactive (publications) or non-reactive (methods), and mutators (methods), something like grapher. Your package unify everyhing in the subscription concept.

Caching is crucial in many apps to ensure you alliviate the server preassure and have smooth experience in your client apps, this is really convenient specially on native experience. I have noted huge impact on my apps by just adopting a cache mechanism for subscriptions or actually any request kind. I remember that on private projects I used ccorcos/meteor-subs-cache, but we had to fork it since was unmaintaned and we required to tweak in order to meet some specific requirements. Currently, I adopted an approach outside Meteor, and specific for React, which is the ahooks/useRequest hook. Many of the common cache mechanisms are already implemented and maintaned there, loading delay, cache, swr, error retry, debouce/throttle, and so on. But this is just a React-only solution and you still need to shape your utilities to use Meteor API.

jam:pub-sub is a specific caching solution for Meteor apps that surely is going to be useful for many projects.

5 Likes

I think at the very least the cache part should be added to Meteor core. Taking this for a spin.

3 Likes

I’ll take some time in the following days to try this out, check out your code, and think of ways to bring this work to the core.

Great work with this one!

2 Likes

I just added a big new feature: Change Streams-based pub/sub :tada:.

Here’s a quick look on how to use it:

Create a publication using Meteor.publish.stream

// server
Meteor.publish.stream('notes.all', function() {
  return Notes.find(); // expects a cursor or an array of cursors just like Meteor.publish
});

Subscribe like you normally would on the client:

// client
Meteor.subscribe('notes.all')

That’s it. By using Meteor.publish.stream , any database writes to the Notes collection will be sent reactively to all connected clients just as with Meteor.publish but you’ll be using MongoDB Change Streams instead of the oplog.

At the moment, I would consider this feature experimental. Based on previous Change Streams experiments, it seems that using Change Streams as a wholesale replacement for the traditional publish / subscribe could “just work”. However, in practice it may be a “Your Mileage May Vary” type of situation depending on the frequency of writes, number of connected clients, how you model your data, and how you set up the cursors inside of Meteor.publish.stream. With that said, if you’re interested in this feature, I’d encourage you to try it out and share your findings.

16 Likes

It would be really interesting to see some benchmark results where an app is taken and tested under the same load with different implementations. Maybe some E2E tests based on cypress or similar. Would be great if we could see some actual hard numbers comparing the standard oplog performance vs your change streams based implementation.

Granted, ensuring that such performance testing is actually accurate and is “apples to apples” is really difficult. But I’d say there’s no need for academic leves of strictness here. Even posting some numbers for just specific apps, without extrapolating wider conclusions from them, would be very interesting.

Does anyone have such an app at hand that relies materially on pub/sub and has test cases already available that put a suitable amount of load on the server? Your input here would be much appreciated.

But in any case, big props to @jam for working on the foundations of Meteor. Very nice!

2 Likes