I am gradually preparing our application to be ready for Meteor 3.0. One of the major conversion steps is that all sync collection functions need to be converted into its async versions at the server side. We extensively used ‘peerlibrary:reactive-publish’, but since this does not support async collection functions, we switched to using ‘meteor-publish-composite’ instead.
We were able to convert the majority of our ‘reactive-publish’ needs with this package, however we still have a few code blocks that we can’t get an alternative solution for. Below is a simplified example of a situation I am not able to solve with e.g. ‘meteor-publish-composite’:
Imagine 2 collections: collectionA and collectionB. Collection B documents may have a reference to a collection A document. I want to publish all collection A documents that are being referenced to by at least 1 collection B document, but I don’t want to publish collection B documents.
With ‘peerlibrary:reactive-publish’ this was easy:
I also use “peerlibrary:reactive-publish” due to its flexibility and promotion of reusable code. I plan to continue using it even on Meteor 3.x, though with adjustments for async adoption. However, I understand the challenges involved, such as migrating all peerlibrary packages from CoffeeScript to JavaScript, incorporating async support into autoruns, and eliminating any fiber usage if necessary.
Considering this, I see why transitioning to ‘meteor-publish-composite.’ A possible implementation could be the following:
This will publish completely the collectionA documents that are linked with collectionB, and only the _id and collectionADoc_id of the collectionB documents. Surely this is not as optimal as with the autorun approach, but not sure if publish-composite can have that expressiveness.
Regarding your mention of supporting async collections operations, I’m unsure of your exact meaning. My understanding is that ‘find’ simply returns a cursor, which should be appropriately resolved and published by ‘meteor-publish-composite’. However, if there are issues I’ve missed due to being less familiar with this library and its changes, we can delve deeper into the matter.
Side note: I’m aware that async support has been added to ‘find’ and ‘children’ after this PR. However, there don’t appear to be tests supporting this implementation. This feature is provided in case additional async behaviors are needed in the ‘find’ functions to return the cursors anyway. Maybe in your example provided, using .countAsync() within find is the use-case example, but with the approach I wrote above is not neccesary.
I also thought about using your suggested ‘meteor-publish-composite’ approach, however collection B can be huge, yielding unnecessary communication/data transfers between server and client. The way it is intended to work is that the client first gets a list of collection A documents that are referenced by any B document, and only after he selects a particular A document he then subscribes to all related B documents.
For the time being, the only efficient solution I can think of is to add an additional ‘docBReferenceCount’ field to collection A documents that is being updated each time a document B reference is added/removed.
But having an updated ‘peerlibrary:reactive-publish’ package would be a very welcome addition to Meteor 3.0, I’d even suggest it should be part of Meteor’s core libraries.
While not the most efficient option, it eliminates a migration step. For smaller databases and less frequent use, it may still be worthwhile. Hopefully you don’t have much collections and data to migrate.
I support on integrating and reviving ‘peerlibrary:reactive-publish’ into the Meteor 3.x ecosystem as it simplifies reactivity on related data. It would be great to include it on the Meteor core in the future if the community is convinced. The way it approaches the solution fits perfectly with Meteor reactivity API.
Well, after using ‘meteor-publish-composite’ for about 2 months now, it turns out this package is not reliably reactively updating all modified documents on the client side in all cases: certain sequences of document updates on the server cause some document updates to be skipped on the client. I have tried to narrow down these situations, but have given up. What I do know: it is not related to using the new async collection functions though, the problem is also there when using sync collection functions.
So for the time being, we have reverted back to using ‘peerlibrary:reactive-publish’ and using sync collection functions, which works reliably, but this makes us being stuck with Meteor version 2.16.
What we really need is an update of ‘peerlibrary:reactive-publish’ for Meteor 3.x, so I hope this will happen any time soon.
P.S. A very odd observation we noticed when reverting back: by just adding ‘peerlibrary:reactive-publish’ in ‘packages’ but still using using ‘meteor-publish-composite’, the problem of skipped client side document updates is almost gone, but still occurs every now and then.
Just wondering if anyone has found a good Meteor 3 substitute or alternative way to achieve what peerlibrary:reactive-publish does since the last post on this thread?
Perhaps there’s a manual way of doing it without the simplicity of peerlibrary:reactive-publish? I still need to do a deep dive to see if I can figure anything out.
As best I can tell, meteor-publish-composite isn’t a very good fit for me since I need to know on the client when each of the related subscriptions is loaded. meteor-publish-composite bundles it all into one subscription.
For my apps, nearly all of the Meteor 3 migration is complete, but publications using peerlibrary:reactive-publish is a blocker. I’m committed to this migration because peerlibrary:reactive-publish uses the most of Meteor core concepts and feel more native than meteor-publish-composite. It enables Tracker’s autorun on the server to track cursors automatically (any ReactiveVar dependency), including derived ones, and publish related documents when they change. Autorun blocks also make code reusable, since server publication logic often mirrors client logic, avoiding separate dependency structures as with meteor-publish-composite.
I’ve got all the original tests of peerlibrary:reactive-publish passing except one. I’m working on that last edge case and plan to add tests for some performance issues I noticed when migrating to async. Besides, I want to verify I don’t break any core behaviors since this package patches core functionality.
This process has been quite an odyssey: the peerlibrary packages involve advanced reactivity handling, interdependencies of packages, and are written in CoffeeScript. I’ve been patiently converting each package to JavaScript and tackling the challenges to make them async-await compatible through deep study. Migrating the tests to TinyTest was also tough. The good news is I’m almost there, and I’ll announce when it’s ready.
I can’t give dates yet, just share the level of commitment I have to this. These days I’m doing a lot of deep work in all sides of my work. On the Meteor core side with the modern bundler and other responsibilities. I just want to keep everything fun and stress-free. Once I’ve completed the final fixes on the new reactive-publish work, I’ll publish a beta for all you.
@nachocodoner This is very good news, I am really looking forward to that!
Currently I am using zodern:relay with reactive pipelines as an alternative to peerlibrary:reactive-publish, this package looked promising too however it occasionally causes fatal server crashes (exceptions caused by unexisting document changes or removes), so I took the package locally and now catch these exceptions so they don’t crash the server anymore. This is not a proper fix but it is the only (temporary) workaround I could found for using reactive publications in Meteor 3.x.
I’m planning to release an alpha soon and test it in my hobby app, since I’m currently migrating it to Meteor 3. This package is the last piece needed for finishing the migration of my app. I’ll share it here once I do some cleanup and refactor the number of packages to reduce the scope and avoid overcomplicating things on publishing. I’ll likely publish an alpha first. A beta needs more effort, as I still have to verify its impact on existing core functionality. I’ll handle that later.
Keep in mind this is moving slowly, as it’s a side project unrelated to Meteor core priorities. I might take my time before releasing an official version, or maybe not, depending on my capacity and motivation, we’ll see. I’d still love to have your feedback on the process leading up to the official one.