Reactive publish using async collection functions

help MongoDB
1

Hi,

I am gradually preparing our application to be ready for Meteor 3.0. One of the major conversion steps is that all sync collection functions need to be converted into its async versions at the server side. We extensively used ‘peerlibrary:reactive-publish’, but since this does not support async collection functions, we switched to using ‘meteor-publish-composite’ instead.

We were able to convert the majority of our ‘reactive-publish’ needs with this package, however we still have a few code blocks that we can’t get an alternative solution for. Below is a simplified example of a situation I am not able to solve with e.g. ‘meteor-publish-composite’:

Imagine 2 collections: collectionA and collectionB. Collection B documents may have a reference to a collection A document. I want to publish all collection A documents that are being referenced to by at least 1 collection B document, but I don’t want to publish collection B documents.

With ‘peerlibrary:reactive-publish’ this was easy:

    Meteor.publish("refCollectionA", function () {
        (<any>this).autorun(function () {
            let referencedCollectionADocs: string[] = [];
            collectionA.find({}, { fields: { _id: 1 } }).forEach((collectionADoc: any) => {
                if (collectionB.find({ collectionADoc_id: collectionADoc._id }, { fields: { _id: 1 } }).count() > 0) referencedCollectionADocs.push(collectionADoc._id);
            });
        
            return collectionA.find({ _id: { $in: referencedCollectionADocs } });
        });
    });

Any suggestions on how to do this without using ‘peerlibrary:reactive-publish’ and being able to use async collection functions?

1 Like
2

I also use “peerlibrary:reactive-publish” due to its flexibility and promotion of reusable code. I plan to continue using it even on Meteor 3.x, though with adjustments for async adoption. However, I understand the challenges involved, such as migrating all peerlibrary packages from CoffeeScript to JavaScript, incorporating async support into autoruns, and eliminating any fiber usage if necessary.

Considering this, I see why transitioning to ‘meteor-publish-composite.’ A possible implementation could be the following:

import { Meteor } from 'meteor/meteor';
import { publishComposite } from 'meteor/reywood:publish-composite';

Meteor.publishComposite('refCollectionA', {
  find() {
    return collectionB.find({ collectionADoc_id: { $exists: true } }, { fields: { _id: 1, collectionADoc_id: 1 } });
  },
  children: [
    {
      find(collectionBDoc) {
        return collectionA.find({ _id: collectionBDoc.collectionADoc_id });
      },
    }
  ],
});

This will publish completely the collectionA documents that are linked with collectionB, and only the _id and collectionADoc_id of the collectionB documents. Surely this is not as optimal as with the autorun approach, but not sure if publish-composite can have that expressiveness.

Regarding your mention of supporting async collections operations, I’m unsure of your exact meaning. My understanding is that ‘find’ simply returns a cursor, which should be appropriately resolved and published by ‘meteor-publish-composite’. However, if there are issues I’ve missed due to being less familiar with this library and its changes, we can delve deeper into the matter.

Side note: I’m aware that async support has been added to ‘find’ and ‘children’ after this PR. However, there don’t appear to be tests supporting this implementation. This feature is provided in case additional async behaviors are needed in the ‘find’ functions to return the cursors anyway. Maybe in your example provided, using .countAsync() within find is the use-case example, but with the approach I wrote above is not neccesary.

2 Likes
3

I also thought about using your suggested ‘meteor-publish-composite’ approach, however collection B can be huge, yielding unnecessary communication/data transfers between server and client. The way it is intended to work is that the client first gets a list of collection A documents that are referenced by any B document, and only after he selects a particular A document he then subscribes to all related B documents.

For the time being, the only efficient solution I can think of is to add an additional ‘docBReferenceCount’ field to collection A documents that is being updated each time a document B reference is added/removed.

But having an updated ‘peerlibrary:reactive-publish’ package would be a very welcome addition to Meteor 3.0, I’d even suggest it should be part of Meteor’s core libraries. :wink:

Thanks for responding!

2 Likes
4

While not the most efficient option, it eliminates a migration step. For smaller databases and less frequent use, it may still be worthwhile. Hopefully you don’t have much collections and data to migrate.

I support on integrating and reviving ‘peerlibrary:reactive-publish’ into the Meteor 3.x ecosystem as it simplifies reactivity on related data. It would be great to include it on the Meteor core in the future if the community is convinced. The way it approaches the solution fits perfectly with Meteor reactivity API.

2 Likes
5

Well, after using ‘meteor-publish-composite’ for about 2 months now, it turns out this package is not reliably reactively updating all modified documents on the client side in all cases: certain sequences of document updates on the server cause some document updates to be skipped on the client. I have tried to narrow down these situations, but have given up. What I do know: it is not related to using the new async collection functions though, the problem is also there when using sync collection functions.

So for the time being, we have reverted back to using ‘peerlibrary:reactive-publish’ and using sync collection functions, which works reliably, but this makes us being stuck with Meteor version 2.16.

What we really need is an update of ‘peerlibrary:reactive-publish’ for Meteor 3.x, so I hope this will happen any time soon.

P.S. A very odd observation we noticed when reverting back: by just adding ‘peerlibrary:reactive-publish’ in ‘packages’ but still using using ‘meteor-publish-composite’, the problem of skipped client side document updates is almost gone, but still occurs every now and then.

6

Hi,

Just wondering if anyone has found a good Meteor 3 substitute or alternative way to achieve what peerlibrary:reactive-publish does since the last post on this thread?

Perhaps there’s a manual way of doing it without the simplicity of peerlibrary:reactive-publish? I still need to do a deep dive to see if I can figure anything out.

As best I can tell, meteor-publish-composite isn’t a very good fit for me since I need to know on the client when each of the related subscriptions is loaded. meteor-publish-composite bundles it all into one subscription.

Thanks,
Graeme

7

I have news: I’m reviving peerlibrary:reactive-publish and some of peerlibrary’s packages with permission from the original author @mitar. I’m doing this alongside my Meteor core duties and as hobby work, spending under half my work time over several months as I migrate my hobby apps to Meteor 3 (yes, some of us are still in the process!).

For my apps, nearly all of the Meteor 3 migration is complete, but publications using peerlibrary:reactive-publish is a blocker. I’m committed to this migration because peerlibrary:reactive-publish uses the most of Meteor core concepts and feel more native than meteor-publish-composite. It enables Tracker’s autorun on the server to track cursors automatically (any ReactiveVar dependency), including derived ones, and publish related documents when they change. Autorun blocks also make code reusable, since server publication logic often mirrors client logic, avoiding separate dependency structures as with meteor-publish-composite.

I’ve got all the original tests of peerlibrary:reactive-publish passing except one. I’m working on that last edge case and plan to add tests for some performance issues I noticed when migrating to async. Besides, I want to verify I don’t break any core behaviors since this package patches core functionality.

image
image1908×1729 154 KB

This process has been quite an odyssey: the peerlibrary packages involve advanced reactivity handling, interdependencies of packages, and are written in CoffeeScript. I’ve been patiently converting each package to JavaScript and tackling the challenges to make them async-await compatible through deep study. Migrating the tests to TinyTest was also tough. The good news is I’m almost there, and I’ll announce when it’s ready.

I can’t give dates yet, just share the level of commitment I have to this. These days I’m doing a lot of deep work in all sides of my work. On the Meteor core side with the modern bundler and other responsibilities. I just want to keep everything fun and stress-free. Once I’ve completed the final fixes on the new reactive-publish work, I’ll publish a beta for all you.

7 Likes
8

This is so cool! Awesome work!

3 Likes
9

@nachocodoner This is very good news, I am really looking forward to that!

Currently I am using zodern:relay with reactive pipelines as an alternative to peerlibrary:reactive-publish, this package looked promising too however it occasionally causes fatal server crashes (exceptions caused by unexisting document changes or removes), so I took the package locally and now catch these exceptions so they don’t crash the server anymore. This is not a proper fix but it is the only (temporary) workaround I could found for using reactive publications in Meteor 3.x.

1 Like
10

It’d be awesome if we can integrate these ideas into the core.

11

Fantastic, thank you @nachocodoner

12

Hi Nacho,

I just thought I’d ask if you’re any closer to releasing a beta for peerlibrary:reactive-publish?

I think I’d even be happy to go with an alpha version if it passed my own basic tests :slightly_smiling_face:

Regards,
Graeme

1 Like
13

Update: I’ve passed the last broken test, fixed flaky ones, reworked some, and added more coverage to check basic performance and stability.

image
image3168×2299 587 KB

I’m planning to release an alpha soon and test it in my hobby app, since I’m currently migrating it to Meteor 3. This package is the last piece needed for finishing the migration of my app. I’ll share it here once I do some cleanup and refactor the number of packages to reduce the scope and avoid overcomplicating things on publishing. I’ll likely publish an alpha first. A beta needs more effort, as I still have to verify its impact on existing core functionality. I’ll handle that later.

Keep in mind this is moving slowly, as it’s a side project unrelated to Meteor core priorities. I might take my time before releasing an official version, or maybe not, depending on my capacity and motivation, we’ll see. I’d still love to have your feedback on the process leading up to the official one. :blush:

4 Likes
14

Thanks a lot for that. I’ll take anything, alpha, beta, whatever, to test it, this is the last bit that I need to upgrade a 7 year old behemoth.

2 Likes
15

Getting rid of Fibers on the server side is a win, but not having signals and effects on the server side is a bigger loss in terms of code quality.

We need signals and effects on both sides, for clean maintainable easily-refactorable reactive code on both sides.

16

I’ve released an alpha of reactive-publish, as mentioned in this post: Reactive-Publish for Meteor 3: Revived and Ready for Composite Data Publishing

Please test it and share your feedback so we can publish the official version.

Let’s all be on Meteor 3!