Reactive publish using async collection functions

help MongoDB
1

Hi,

I am gradually preparing our application to be ready for Meteor 3.0. One of the major conversion steps is that all sync collection functions need to be converted into its async versions at the server side. We extensively used ‘peerlibrary:reactive-publish’, but since this does not support async collection functions, we switched to using ‘meteor-publish-composite’ instead.

We were able to convert the majority of our ‘reactive-publish’ needs with this package, however we still have a few code blocks that we can’t get an alternative solution for. Below is a simplified example of a situation I am not able to solve with e.g. ‘meteor-publish-composite’:

Imagine 2 collections: collectionA and collectionB. Collection B documents may have a reference to a collection A document. I want to publish all collection A documents that are being referenced to by at least 1 collection B document, but I don’t want to publish collection B documents.

With ‘peerlibrary:reactive-publish’ this was easy:

    Meteor.publish("refCollectionA", function () {
        (<any>this).autorun(function () {
            let referencedCollectionADocs: string[] = [];
            collectionA.find({}, { fields: { _id: 1 } }).forEach((collectionADoc: any) => {
                if (collectionB.find({ collectionADoc_id: collectionADoc._id }, { fields: { _id: 1 } }).count() > 0) referencedCollectionADocs.push(collectionADoc._id);
            });
        
            return collectionA.find({ _id: { $in: referencedCollectionADocs } });
        });
    });

Any suggestions on how to do this without using ‘peerlibrary:reactive-publish’ and being able to use async collection functions?

1 Like
2

I also use “peerlibrary:reactive-publish” due to its flexibility and promotion of reusable code. I plan to continue using it even on Meteor 3.x, though with adjustments for async adoption. However, I understand the challenges involved, such as migrating all peerlibrary packages from CoffeeScript to JavaScript, incorporating async support into autoruns, and eliminating any fiber usage if necessary.

Considering this, I see why transitioning to ‘meteor-publish-composite.’ A possible implementation could be the following:

import { Meteor } from 'meteor/meteor';
import { publishComposite } from 'meteor/reywood:publish-composite';

Meteor.publishComposite('refCollectionA', {
  find() {
    return collectionB.find({ collectionADoc_id: { $exists: true } }, { fields: { _id: 1, collectionADoc_id: 1 } });
  },
  children: [
    {
      find(collectionBDoc) {
        return collectionA.find({ _id: collectionBDoc.collectionADoc_id });
      },
    }
  ],
});

This will publish completely the collectionA documents that are linked with collectionB, and only the _id and collectionADoc_id of the collectionB documents. Surely this is not as optimal as with the autorun approach, but not sure if publish-composite can have that expressiveness.

Regarding your mention of supporting async collections operations, I’m unsure of your exact meaning. My understanding is that ‘find’ simply returns a cursor, which should be appropriately resolved and published by ‘meteor-publish-composite’. However, if there are issues I’ve missed due to being less familiar with this library and its changes, we can delve deeper into the matter.

Side note: I’m aware that async support has been added to ‘find’ and ‘children’ after this PR. However, there don’t appear to be tests supporting this implementation. This feature is provided in case additional async behaviors are needed in the ‘find’ functions to return the cursors anyway. Maybe in your example provided, using .countAsync() within find is the use-case example, but with the approach I wrote above is not neccesary.

1 Like
3

I also thought about using your suggested ‘meteor-publish-composite’ approach, however collection B can be huge, yielding unnecessary communication/data transfers between server and client. The way it is intended to work is that the client first gets a list of collection A documents that are referenced by any B document, and only after he selects a particular A document he then subscribes to all related B documents.

For the time being, the only efficient solution I can think of is to add an additional ‘docBReferenceCount’ field to collection A documents that is being updated each time a document B reference is added/removed.

But having an updated ‘peerlibrary:reactive-publish’ package would be a very welcome addition to Meteor 3.0, I’d even suggest it should be part of Meteor’s core libraries. :wink:

Thanks for responding!

1 Like
4

While not the most efficient option, it eliminates a migration step. For smaller databases and less frequent use, it may still be worthwhile. Hopefully you don’t have much collections and data to migrate.

I support on integrating and reviving ‘peerlibrary:reactive-publish’ into the Meteor 3.x ecosystem as it simplifies reactivity on related data. It would be great to include it on the Meteor core in the future if the community is convinced. The way it approaches the solution fits perfectly with Meteor reactivity API.

1 Like