Apologies if this has been discussed and somehow I missed it, but…
I have been following what I think is the entire discussion, on GitHub and here, since the Fibers note of obsolescence was brought to attention, and it is not clear to me: starting with Meteor 3.0, will the frontend packages also only support async methods?
This is important in the case of Minimongo, because it adds a lot of work when migrating. Minimongo support for a synchronous API, something like findSync, fetchSync, countSync, etc. would be tremendous.
Piggybacking on the post (and thank you to the OP for allowing me to join in ).
I’ve been pondering a potential path for the Meteor 3.0 transition. It’s intriguing to consider that Node 16 (perhaps 18 too) could be patched to maintain Fibers support, albeit with some performance implications.
Since Meteor uniquely bundles its own Node version, has the team contemplated including a Fibers-supportive Node variant in Meteor v3? Such an approach could smooth the transition to Meteor v3, especially for large-scale projects deeply integrated with Fibers.
I imagine a scenario where Fibers usage triggers a gentle deprecation warning, allowing developers a reasonable timeframe to adapt. Every aspect migrated away from Fibers will also benefit from improved performance further encouraging the rewrite.
For extensive projects like mine, the prospect of moving away from Fibers is a formidable one, raising serious questions about resource allocation. The choice between upgrading to Meteor 3 or, regrettably, considering alternatives, is not straightforward.
A phased approach, extending over several months or even a year, would undoubtedly ease this transition. It’s about offering a feasible pathway for developers to evolve their applications with Meteor’s growth, maintaining the community’s strength and commitment to this excellent framework.
The plan for the client has changed many times. As far as I know, the latest plan is to leave the sync api as is for now on the client. Though I saw this on a slack thread a couple months ago so it would be nice to have it confirmed somewhere more public.
@theosp thanks for writing that. I had similar concerns a year ago.
Someone in the community was actively working on using fibers with newer Node versions using CORO_PTHREAD, though there were some issues with Windows and m1 at the time in addition to lower performance.
The problem was v8 removed the easy way for fibers to access a certain piece of data needed in the more performant and reliable fiber implementations. There is another way to access it, but it would be non-trivial and might be less stable between v8 versions. As far as I know, no one has seriously looked into this route.
Now that Meteor is maintaining its own version of Node 14 and deployment tools have or are in the process of updating to support it, it might be more reasonable to have a patched version of Node 20. This would also give us the option of fixing using async hooks with fibers.
There is also the option of submitting a change to v8, but it is a little late to start now and there is no guarantee it would be accepted.
One specific idea is to have a Meteor 2.15 release with:
node 20, patched to support fibers and for fibers to work with async hooks
api compatible with Meteor 3 (doing this would also help identify places in Meteor 3 where it could be changed to provide a migration path or where there are unnecessary breaking changes)
some exceptions that would probably be Meteor 3 only would be top level await (or we could use a non-spec compliant version and document the subtle differences with Meteor 3) and using express in webapp, and api’s that Meteor 3 removed would still be in 2.15.
follow Node’s experience for deprecated functions (log the first time it is used, and provide options to log a stack trace and log every time it is used)
Hey!
For the front end, nothing will change, and the methods with their *Async counterpart should not be affected when 3.0 hits.
For the server, it was decided to remove the non-*Async methods, 3.0 using these methods will result throw an error
Thank you for sharing your idea on a potential Meteor 2.15 release.
I’m curious about your involvement with the Meteor project or your expertise in this area. Your proposal sounds like it requires a deep understanding of Meteor’s infrastructure to undertake. Are you closely connected with the Meteor development team, or do you possess the technical know-how to implement such changes?
Also, I’m interested in learning about the practical steps that could be taken to realize this idea. What kind of collaboration or resources would be necessary to make this a reality?
Your insight into this is very intriguing, and I look forward to understanding more about your perspective and the feasibility of this proposal.
If it helps, there was quite an extensive discussion, in which even Ben Newman from the original MDG team got involved, detailing the challenges. In short, it’s nigh impossible to do it as reliably and as performantly as it used to be pre-Node v16. Here’s the discussion: async_hooks & node-fibers discussion · Issue #365 · laverdet/node-fibers · GitHub
Things might have changed with newer Node and v8 versions, but at the time there were two issues:
v8 broke how fibers archived threads, needed by the more performant backends. node-fibers has multiple backends, and at least one could still work with newer node versions
node-fibers wasn’t fully compatible with async hooks
Both issues could be fixed with a custom build of Node that exposes the necessary internals from v8 and async hooks. The link @illustreets shared about async_hooks is from Node 8, but the situation is probably more complicated now as async hooks have matured and are used more in Node’s internals.
There are downsides to using a custom build of Node, but since Meteor 2 has already been for a few months I’m not sure if they matter as much anymore.
@theosp regarding the feasibility of my suggestion, I’m not sure. It would be up to Meteor Software if they wanted to create and maintain a second custom build of Node and were willing to divert some resources from Meteor 3 to release Meteor 2.15. Most of the discussions and research around fibers is from Node 16, so it is possible something else has changed since then that makes it more difficult or impossible.
If the purpose is to support transitioning to Meteor 3, is the current setup not enough?
Slowly convert to async while in 2.x and then jump to 3.x when ready.
So instead of trying node-fibers to work in current node versions, why not just extend node 14 support longer for bigger projects to transition?
About adding support for node-fibers in Meteor 3.0
this might be a lot of effort now wherein months (years?) of development has been dedicated to removing it
this is an opportunity to let go of legacy code that is holding back Meteor’s advancements
About adding support for node-fibers in Meteor 2.x to support new version of node:
if this will be done, I hope it will not affect the development of Meteor 3.0 (but since the resources look limited, any development to other things will have an effect)
it will still be 2.x and not Meteor 3.x. So why not just extend Node 14 support for transition purposes?
If Meteor in 3.0 would somehow continue reliance on fibers, then aside from technical merits one major problem would be just the fact that the wider js ecosystem is just not used to sync code like that. This would be a big problem when getting new people to work on Meteor projects. For that alone, it seems it would not be a good direction, assuming that async/await can actually be made to work.
Sorry, I didn’t spend enough time thinking about my first two comments before I wrote them, or reading @theosp’s comment thoroughly enough.
I agree that supporting fibers in Meteor 3 isn’t a viable option.
I think this part of @theosp’s post makes a very good point:
This might be more of a communication issue rather than a technical issue. It’s getting better, but there has been a lack of clear, official information on Meteor 3, which might have made this less straightforward for projects. Also, the current docs for preparing for Meteor 3 imply that the client also needs to migrate to async, which is mostly unnecessary and might mislead people in how difficult it would be to upgrade to Meteor 3.
My suggestion for Meteor 2.15 would also give apps more time to prepare for Meteor 3 while using a newer Node version, but there probably would be significant opportunity costs with it and Node 14 ESM probably is good enough. There is a limit to how long Node 14 ESM will be practical - over time an increasing number of npm packages will no longer release security updates for Node 14, and at some point the dependencies used by Node 14 will also reach end of life and no longer receive security updates. Hopefully either one being a serious problem is far enough in the future.
There is one part of the suggestion for Meteor 2.15 I think would be beneficial on its own: by default log the first time a deprecated api is used to ensure Meteor developers are aware of the deprecation and that the api is being used.
I know what you mean. It looked scary at first sight. Our product is by most measures also complex - as an indication, we have our own 100+ internal packages, besides everything else. Then, on top of MongoDB, we communicate with several internal services over HTTP. All this was written using the old Meteor “synchronous” approach.
What I hope may be good news for you is that it works quicker than expected, at least on the server side. We are over 50% of the way to rewriting everything using async and await, and it has been a rather painless process. Of course, it depends on the IDE you use, so it can highlight when you are calling async functions without prefixing them. Also, depends on how modular your code is, how many tests. But there are parts of our code base that are a bit challenging in that respect, and those, if anything, benefited from a bit of tidying up.
No way that, in our case, a complete rewrite would have been faster. Conversion, so far, has taken roughly 200 dev hours. Frontend, on the other hand, is a totally different matter. Also, isomorphic code that relies on Mongo will be a thorn on the side, especially in third party packages that now need to be forked. But nothing insurmountable if you ask me.
We need to keep in mind that more and more npm packages will rely on new Node.js APIs incompatible with the Fibers approach, so keeping Fibers running for longer will not help Meteor in the long term. The best example of these problems is APM agents (like Datadog).
Meteor 2.x is a good alternative for people who need to run Fibers code longer. Still, ESM will be impossible in a few months as security fixes are now targeting a very distant version of Node.js.
So IMO what we have right now is the best possible scenario considering all the variables:
Meteor 2.x with ESM support until it’s not possible (already with Async APIs available);
Meteor 3 focuses on the future (running Node.js 20).
For those uncertain about the future, please ask your questions in an issue on GitHub, and we can submit PRs to have the answers in official docs.
To start, I hope Meteor moves away from using its own semantics and follow the broader JS community. For someone unfamiliar, dropping support for ESM in Meteor 3.x would be a crazy move.
Anyway, I think communication is critical, and I hope the core team learn the historical lessons from MDG. Lots of a great work and planning is going to Meteor 3.0, last thing they want to is blunder the communication.
).