Reactive-Publish for Meteor 3: Revived and Ready for Composite Data Publishing

Hello everyone!

Great news for those still migrating to Meteor 3 and blocked by peerlibrary:reactive-publish.

nachocodoner:reactive-publish is the modern version with async support. This package is an alternative to reywood:publish-composite. Feel free to use this solution to publish composite data.

I decided to revive it, since I use it in my Meteor 2 hobby apps. With the core updated and modern tools arriving, I found time to move my apps to Meteor 3 over last months. The last hurdle was peerlibrary:reactive-publish. You can read about why I keep using it here.

The package is now in release candidate and the full test suite passes :white_check_mark:. I’ve integrated it into my apps, and all tests are green. I’ll release a beta once I confirm that Meteor core tests aren’t affected by the changes to core publish and other modules, but so far both package tests and my app tests pass.

I’d appreciate feedback from anyone who needs it or wants to try it. Are your tests passing? I published early to address the requests from Reactive publish using async collection functions.

Hands on

meteor add nachocodoner:reactive-publish@1.0.0-rc.1

Basic usage

The package makes any changes in your queries inside an autorun block publish updated data reactively, even for linked collections.

Because of isomorphism and minimongo, you can reuse the same code on client and server. This keeps your query logic unified, one of the main benefits over publish-composite.

Meteor.publish('subscribed-posts', function () {
  this.autorun(async () => {
    const user = await User.findOneAsync(this.userId, {
      fields: { subscribedPosts: 1 },
    });

    return Posts.find({ _id: { $in: user?.subscribedPosts || [] } });
  });
});

In the example above, you publish the user’s subscribed posts. When the User’s subscribedPosts field changes, autorun reruns and publishes the updated posts. Any queries with related data work the same way. You can also publish an array of cursors and use the same logic as in a normal publication body.

The package and more details are on GitHub: nachocodoner:reactive-publish

Roadmap

Current version: 1.0.0-rc.1. I’ll update this post with future releases.

  • Stability :white_check_mark:

    • Ensure core changes in this package don’t affect Meteor core tests
    • Release betas and RCs, with a feedback period for early adopters
  • Expansion

    • Support for AsyncTracker and ReactiveVarAsync on the client
    • Migrate peerlibrary:subscription-data to support publishing derived data reactively
  • Performance


:comet: Keep going with Meteor 3 and building cool apps!

13 Likes

I’d be really intrigued if it beats reywood:publish-composite in terms of performance. publish-composite API/DX is very grotesque and feels unnatural compared to reactive-publish.

1 Like

Thanks a lot for your work! Works perfectly here with a bunch of filtered published collections, with no changes other than the async/await makeover. Could not see any change in runtime behavior.

1 Like

Awesome! Does that mean that reactivity works on several nested levels? This was the main drawback with reywood:publish-composite for me, since reactivity only worked on top level.

2 Likes

Works fine for my app, tx very much @nachocodoner :pray:

1 Like

Anything inside autorun runs reactively, so you can prepare cursors for multiple collections depending on related fields got from Mongo operations in the same autorun body.

Does that answer your question? I’m not very familiar with the internals of publish-composite, but I think you mean the limits of nested relationships there? in reative-publish you programatically describe all relationships and cursor without any limits.

1 Like

That’s pretty cool, thanks! Yes, publish-composite only reacts on changes on the top-most query.

1 Like

Published new version: nachocodoner:reactive-publish@1.0.0-alpha.5

  • Fixed a reported issue where reactive and non-reactive publications conflicted. #2. Thanks @graemian for the detailed reproduction.

Report me any issue you find!

I will work next on pass the Meteor core tests, so I can verify full stability of the package with Meteor, and release the first beta.

5 Likes

Published new version: nachocodoner:reactive-publish@1.0.0-rc.1

I published the RC version 1.0.0-rc.1. This release includes the final stability fixes based on Meteor core tests.

I’ll keep testing the library in production apps before the official release. Feedback from anyone adopting it is welcome, and I’ll address issues if needed. Official release come next after a period of further testing.

After the official 1.0.0, I’ll focus on expanding the library, mainly by adapting peerlibrary:subscription-data to enable reactivity for any derived data. I’m excited about this, as that approach felt powerful for making almost anything reactive. After that, I plan to measure performance across the library to get insights on how it handles reactivity in comparison with other packages achieving the same.

2 Likes

I have a document I wrote in 2023 about my findings on how to publish data with Meteor. I didn’t review it, so it’s 100% Meteor 2, and maybe I’ve already changed my mind about some things, but it could help people decide between composite or reactive-publish.

Again: I didn’t review it now - I just copy-pasted an old document written in a private context and not intended to be shared publicly :wink:

Publications

Publications Official doc

https://docs.meteor.com/api/pubsub.html#Meteor-publish

Examples: https://github.com/quavedev/meteor-blaze-samples/blob/main/docs/publications.md

Queries

Meteor queries return cursors, and these cursors create live queries when returned by publications. Live queries calculate the data returned from oplog to find changes and propagate them to connected clients.

Reuse observers as much as possible

When you create a Live Query, it creates an observer internally. Observers watch changes in the DB and notify the Live Query. However, if there’s an existing observer for a similar query, Meteor won’t create a new observer. (Read more)

This is why you should never create publications that receive arguments that change all the time, like a date with the current time. When you do this, it’s recommended to use a method or refactor your schema to represent this as a state instead of a date.

Arrow functions on publications

Don’t use arrow functions in publications because it’s very common to get the userId or other things from the this publication context.

Get currently logged user

Don’t use Meteor.userId() on publications. Instead, use this.userId and Meteor.users.findOne(userId).

This is a Meteor limitation. Meteor.userId() is allowed anywhere except publications.

Return empty data

To return empty data, return a cursor or an array of cursors. If a publish function doesn’t return anything, it’s assumed to be using the low-level this.added, this.changed, this.removed interface, and it must also call this.ready once the initial record set is complete.

Return custom data and fields based on currently logged user or subscribed parameters

In some cases, limiting the amount of data to publish is crucial because publishing too much unnecessary data causes client-side performance issues. You can do this by limiting the search on the database or limiting the published fields.

Return cursor or array of cursors

Don’t return an array of data like .find().fetch() because publications expect a cursor or an array of cursors as a return.

Meteor.publish("countries", function (filter = {}) {
  const userId = this.userId;
  if (!userId) {
    return [];
  }
  const user = Meteor.users.findOne(userId);
  if (!user) {
    return [];
  }
  if (user.username.startsWith("user1")) {
    return CountriesCollection.find({ name: { $regex: /^a.*/i }, ...filter });
  }
  if (user.username.startsWith("user2")) {
    return CountriesCollection.find(
      { name: { $regex: /^b.*/i }, ...filter },
      {
        fields: {
          name: 1,
        },
      }
    );
  }

  return CountriesCollection.find({ ...filter });
});

You can also return multiple cursors. This approach should be used instead of multiple publications that are usually used together:

Meteor.publish("usersData", function ({ usersIds }) {
  return [
    UsersCollection.find({ _id: { $in: usersIds }}),
    UserPreferences.find({ userId: { $in: usersIds }})
  ];
});

This is good because of collocation benefits. Since they’re used together, it’s beneficial for all developers to have the same code in the same publication. Usually changes in one query should affect other queries if they’re always used together.

Related Reactive Data

When we have related data, sometimes a change in one document could lead to different publications from another collection.

These solutions are used to avoid multiple round-trips to the server to ask for related data.

For example, if you have a Campaign with CampaignAttachments, you could have a publication like:

Meteor.publish("campaignDataV1", function ({ campaignId }) {
  return [
    CampaignsCollection.find({ _id: campaignId }),
    CampaignAttachmentsCollection.find({ campaignId })
  ];
});

But now let’s think that we don’t want to have campaignId in an attachment. For some reason we have different data modeling, like an array of attachment ids inside Campaigns (I think the approach above is better for this case but here’s just an example) maybe because Attachments are used in different places. So let’s see how this publication would look:

Meteor.publish("campaignDataV2", function ({ campaignId }) {
  const { attachmentsIds } = CampaignsCollection.findOne({ _id: campaignId }, { projection: { attachmentsIds: 1 }});
  return [
    CampaignsCollection.find({ _id: { $in: [campaignId] }}),
    AttachmentsCollection.find({ _id: { $in: attachmentsIds }})
  ];
});

Now let’s think about how the Attachments cursor will behave when this attachmentsIds changes. What do you think? Yes, you’re right. Nothing is going to change because the array is not reactive. The body of a publication is not reactive, only the cursors returned by the publication.

Let’s discuss three solutions here:

  1. Data modeling

The first solution would be to use a data model that’s better structured for this publish. In this case, the first publication code would be better campaignDataV1, but in some cases we can’t change the data model so we also have other options.

Usually it’s a bad idea to store arrays inside documents when these arrays can grow based on usage. This attachmentsIds model is an example of bad usage.

  1. Publish composite

We can also compose publications. The best way to think about this is thinking in a tree. If the tree is small (2 or 3 level depth), this is a good choice and we have a package for that. See an example:

import { publishComposite } from 'meteor/pathable:publish-composite';
publishComposite("activeCampaignListDataV2", function ({ teamId }) {
  return {
    find() {
      return CampaignsCollection.find({
        status: CampaignStatus.ACTIVE.name,
        teamId,
      });
    },
    children: [
      {
        // this is the campaign document retrieved from the find above
        // each document will cause a call for this `find` in the children array
        find({ attachmentsIds }) {
          return AttachmentsCollection.find({ _id: {$in: attachmentsIds},
            status: AttachmentStatus.PUBLISHED.name, });
        },
      },
    ],
  };
});

And why is this bad for big trees? Any change is going to fire many cursors being recreated below this level in the tree, causing load in MongoDB.

  1. Reactive Publish

For big trees it’s better to use another approach. Think if you could have a block of autorun inside the server for a publish function. Yes, with this package you can:

// peerlibrary:reactive-publish
Meteor.publish("activeCampaignListDataWithImagesV2", function ({ teamId }) {
  this.autorun(() => {
    const campaignsCursor = CampaignsCollection.find({
      status: CampaignStatus.ACTIVE.name,
      teamId,
    });

    const attachmentsIds = campaignsCursor
            .fetch()
            .flatMap(({ attachmentsIds }) => attachmentsIds);
    
    const attachmentsCursor = AttachmentsCollection.find({ _id: {$in: attachmentsIds},
      status: AttachmentStatus.PUBLISHED.name, });
    
    const imagesCursor = ImagesCollection.find({ attachmentId: {$in: attachmentsIds} });

    return [campaignsCursor, attachmentsCursor, imagesCursor];
  });
});

So even in big trees you can have a constant number of queries using this package and still keep the change of documents causing a rerun of the other finds.

Another example comparing these approaches.

In this case, when using this.autorun, the return of the main function is not used, just the return inside the autorun block.

Avoid find without projection in the server

Always select only needed fields on searches if you don’t need all fields.

This is important because it reduces performance issues like database server load, Node.js memory usage, and network traffic.

In most cases you should already destructure the fields so it’s clear that the whole object is not returned.

This makes no sense in the client as MiniMongo stores data directly in IndexedDB and no network is involved in the process.

Be careful with cursors that are returned from publications as it can be hard to know all the fields that are used in the client, but when possible you should do field selection (projection) also in published cursors.

// FIND ONE
// BAD: Load full document but uses just name
const campaign = Campaigns.findOne({ _id: campaignId });
console.log(campaign.name);

// GOOD: Load only what you need
// It's better to avoid `campaign` as it may appear that you have the whole object
const { name } = Campaigns.findOne({ _id: campaignId }, { projection: { name: 1 } });
console.log(name);

// FIND
// BAD: Load full document but uses just crmTriggers
const campaigns = Campaigns.find({ teamId }).fetch();
const crmTriggersByCampaign = campaigns.map(campaign => {
  // do something and return
  return transform(campaign.crmTriggers);
});

// GOOD: Load only what you need
// It's better to avoid `campaign` as it may appear that you have the whole object
const campaignsWithCrmTriggers = Campaigns.find({ teamId }, { projection: { crmTriggers: 1 } }).fetch();
const crmTriggersByCampaign = campaignsWithCrmTriggers.map(({ crmTriggers }) => {
  // do something and return
  return transform(crmTriggers);
});
6 Likes

Good points, thanks for sharing. As you stated:

  • publishComposite is better for simple or small hierarchies.
  • reactive-publish helps with large datasets or deep relationships, avoiding exponential queries.
  • reactive-publish loads more efficiently but is less granular in reactivity, since all queries rerun together.

In theory this holds and makes sense. Implementation details on each package might still cause performance mismatches in real scenarios. Though, IO is often the slowest part, so the conclusions likely stand. I’m excited to run benchmarks to confirm and explore optimizations where possible, since reactive-publish also supports granular reactivity with nested autoruns or composed ones, which when used properly can reduce unnecessary query re-runs on triggered changes.

It’s great that Meteor 3 brings back both packages!

2 Likes