Thingking about Meteor 2.0 šŸ¤”

core
101

Iā€™m not familiar with Meteor internals, but prima facie I expect it to be quite a major work, almost a rewrite of all code thatā€™s coupled with fibers. Thatā€™s why I made the suggestion only in the context of Meteor 2.0.

Every API that hides the asynchronous execution would have to be changed to be Promise-based. So

const user = Meteor.users.findOne();

might become

(async () => {
  const user = await Meteor.users.findOne();
})(); // framework may support top level `await` but let's skip the nitty-gritties iin this discussion

or its equivalent:

Meteor.users.findOne().then(user => {})

Hopefully there would be a way to make the current fibers approach an opt-in, maybe via a package. Somebody familiar with how fibers are used internally might be able to tell how backward compatible that could be made. But if thatā€™s not feasible, it shouldnā€™t stop this evolution; then Meteor 2.0 could be a recommendation for new apps while current apps could stay with 1.x and gradually migrate if worth it for that individual project.

Well, ā€œmanyā€ might be subjective, but I think there are enough to warrant a revisit of the decision. More importantly itā€™s the nature of those issues. And perhaps even more importantly, any such problem is doomed to be specific to Meteor; the rest of the Node.js community is unlikely to find any incentive in fixing them. See the example I shared earlier.

Today, this problem can be considered ā€œsolvedā€ in Node.js because of async/await. Since Meteor is not a language, but rather a framework on Node.js, it would be better to defer to Javascript/Node.js standards where feasible ā€“ if we were to apply the principle of least astonishment here. A newcomer is more likely to read about Node.js first than Meteor.js, right?

Sure thatā€™s the decision to be made, and I hope the points Iā€™m putting forth would help with that. I have come to conclude that while adopting fibers was likely a good decision at the time, it has become a technical debt considering the current state of Node.js and its ecosystem, and of course also considering the current state of Meteor.js.

I donā€™t think he said coroutines is ā€œbetterā€ than async/await, just that with coroutines we wonā€™t need those separate keyword. But forward to today, we have those separate keywords as part of the language itself! And coroutines may never become a standard in JS (for good reasons).

Iā€™ve explained my take from his presentation earlier; pasted below is the part that addresses the specific points you raise here:

My point is not whether coroutines are ā€œbetterā€ as a concept. My simple submission is: is Meteor still compelled to live with the tradeoffs of using fibers? If Meteor were to be written from the ground up today, would one have still used fibers? I reckon that removing fibers would not only eliminate the technical debt whose interest has to be paid in the form of the kind of issues I listed earlier, but also help with the adoption and community contribution.

2 Likes
102

Yes, but I only had to use wrapAsync for third party API the rest of our code is all sync which is easier to read. So 95% of our code read sequentially (all the DB calls) thanks to Fiber and the 5% (even less in our case) when we call third party API we just wrap it in a function.

Well it depends, I came from Java and the sequential nature of Meteor appealed to me, I agree with the folks from Kotlin, I do think itā€™s the better and more natural way to handle async.

I understand you point/concern @gaurav7 that Meteor is diverging from how async is now being handled in a typical modern node backends and yes we might end up in some niche issues, issue (#9796) is the most interesting, which turned out to be V8 defect on how threads being managed. But I think given the history, available resources, and even tradeoffs (there are strong arguments for using coroutines to handle async), I personally think the effort/discussion for Meteor 2.0 should be focused somewhere else but anyway I think thatā€™s the core maintainers call, since theyā€™re impacted the most by this refactor, so I rest my case :slight_smile:

1 Like
103

Agreed. I think the biggest impact of removal of fibers would be to make the maintainersā€™ lives easier once the feat is accomplished; then they wonā€™t have to deal with all the overhead :smile:

1 Like
104

We should at least stop depending on Fibers when it comes to the new packages. Maybe over time we can refactor some of our most valuable packages to become async / await. For me it might be cool to discover just exactly how Minimongo works while adding a new feature hydration. :slight_smile:

105

Iā€™m happy that at least some initial action came out of this. If this can be attributed or if it is just timing coincidence.

That is what this thread is about. Major release allows for breakages like these to take place.

Very interesting debate here around Fibers. Just a thought, maybe we could figure out to do something different with them. Some improvement that would push it to new level instead of a removal? Or some way to make things more compatible with await/async while keeping with Fibers? Or the other way around.

Though in the end it comes to what would be the goal with 2.0. Is it to lower the learning curve or is it to merge more with Node.js community? If Meteor is heading to be completely on npm, then Fibers might be a show stoppers for some developers from other projects. Going fully to npm is a major rework anyway.

Thanks @gaurav7 for the code sample. Brings things into perspective. I have to admit that I very much like the Meteor way and it would be great if we could figure out how to keep it as much as possible in case of removal of Fibers, while allowing for all the other stuff (plus keeping backward compatibility).

1 Like
106

Its timing, but this topic did inspire me to contribute more on the core side. Iā€™m now in the progress to investigate what, how and when to contribute to Meteor.

2 Likes
107

Its never about extremes right? In the end, Meteor is a set of packages in one mono-repo. We could for example refactor packages one by one and label them as major update when they are not relying on Fibers anymore. In the end we just need an agreement on where to go. Ofcourse we need some Core Devs approval on this major effort.

2 Likes
108

@gaurav7

Letā€™s come to the reasons we use Meteor.wrapAsync (or Future like we do directly).

First, in a server-side method, you have to tell the framework how it can send the result back to the user (i.e. the client). An asyncCall by itself is not enough as your callback function is not in the main thread of the server-side method. I suspect other frameworks use something similar.

The second reason which is within the code for Meteor.wrapAsync you will notice that the callback is wrapperd in Meteor.bindEnvironment ā€“ it binds the callback to the current environment. You need this if within your callback you are making DB changes (otherwise itā€™s not needed).

You canā€™t get around either of these in ANY framework. So the question is: How can we make it transparent for non-Meteor users to lessen the learning curve (and adoption barrier).

The trend right now is using Promises. Or, if there is a way within the Meteor compiler to use async/await within the server-side method, then great. I donā€™t know enough about JS compilers to make a statement.

So to answer the question, I would opt for an internal implementation of Promise to get around this (a bit like the AWS SDK as mentioned above)

109

@ramez I do understand why Meteor.wrapAsync and Future (and similarly Meteor.bindEnvironment and meteor/promise) are used in Meteor today. This thread has become quite long and maybe you didnā€™t get a chance to read my earlier comments, but I appreciate why these decisions were taken by Meteor. The crux of my argument is only that the time has come now for Meteor to revisit that decision, keeping in mind the current state of Node.js and its ecosystem ā€“ just like supporting NPM packages was a necessity to keep Meteor relevant within the Node.js landscape, and it materialized in v1.3.

Btw, if I understood these points correctlyā€¦

ā€¦it seems youā€™re not considering that Meteor already supports returning a promise from server-side method? E.g.

Meteor.methods({
  uploadToS3() {
    return s3.upload({}).promise();
  },

  async uploadToS3AndLog() {
    const response = await s3.upload({}).promise();
    console.info('s3 upload response:', response);
    return response;
  }
});

ā€¦ both methods return the response to the client-side, after calling the s3 api asynchronously.

If more and more external libraries support promises, maybe Meteor developers would need to use less and less of fibers-specific helpers.

But App-Developer Experience (that the above example relates to) is not the only reason to remove fibers, nor the most important one. There are multiple reasons why removing fibers seems like a must-do to me, even if it calls for a backward-incompatible v2.0. Let me rank them here by the quantum of positive impact:

  1. Meteor would not only get rid of all known fibers related bugs and incompatibilities (see: comment-44 and comment-90), but also the unknowns (nobody knows how many of them are waiting to be discovered), and also the ones that might get introduced in the future (such has been the unfortunate history of Meteor with fibers).
  2. Not having to maintain a non-native paradigm of asynchronous programming would free up the already few maintainers to focus on things that matter.
  3. The absence of this alien concept (alien to the majority of Node.js developers) would encourage more developers to adopt Meteor, and consequently more library authors and tool developers.
7 Likes
110

I had a quick look at how everything is bundled. Apparently boot.js runs the individual packages within a Fiber


ā€¦ and I guess itā€™s up to the individual APIs to call Fiber.yield (or any other method that does something similar).

So I believe it might be possible to refactor each package and tell the bundler to not run those packages inside a Fiber. If this path could be taken, then maybe this exercise would not need a big-bang change and it might also be easier for the community to contribute. But I canā€™t say for sure, since I am not aware of the implicit coupling between the packages and the bundler. Also not sure if this can be done in a backward-compatible way.

Though I think Iā€™m satisfied enough to at least propose this on Github and then leave it to the wisdom of Ben and co; shall do so soon.

3 Likes
111

Thatā€™s not quite true. The Promise is resolved in the method and the (normal) result is returned. As it happens, Meteor uses Fibers to achieve this behaviour.

112

@robfallows I meant the same :slight_smile:.

Not sure how I miscommunicated to you, perhaps I should have also shown that the following works:

if (Meteor.isClient) {
  Meteor.call('uploadToS3AndLog', (uploadResponse, err) => {
    // `uploadResponse` matches `response = await s3.upload({}).promise()` on server-side
  })
}

It appeared to me that @ramez seems to think using Future is the only way to return the result of server-side async operations to client-side, to which I clarified that my example would also work if the server-side async operation is done via promises.

Update: @robfallows ok I see now that what I wrote could be interpreted as the client also receiving a Promise. Like you rightly said, thatā€™s not how if works. The client-side handling remains the same. My focus was on being able to return the response of an async server-side operation using promise

2 Likes
113

You are not sure because I did not state any support or non-supportā€¦ :wink: My intent was to get people to think.

If you are talking about removing Fibers in the next release of Meteor, as someone with a few apps deployed I will not look at it with excitement, due to all the re-work needed if I need to upgrade. The paradigm of asynchronous code is not a natural fit to developers transitioning from pre-Node.js server side platforms and this applies to most corporate/enterprise developers. Server-side Meteor is really easy on these developers, specifically because the async nature on Node is abstracted for them, specially for MongoDB work. I do understand the appeal to people coming from Node.js thoughā€¦

So, I sit on the fenceā€¦ Maybe leaning towards not breaking all the apps out there while Fibers is still usable.

A few old discussions on the subject for reference:

3 Likes
114

Cool, my intention behind saying so was to make you reveal your intent :smiley:

Thanks for sharing the previous discussions on the topic, I should have searched the repo issues harder. Gonna go through them to see if itā€™s worth re-igniting the discussion in the repo.

Btw, Iā€™m not at all suggesting fibers should be removed in the next release. The OP asked about Meteor 2.0 and I thought that would be the best time to revisit a fundamental decision like fibers considering todayā€™s realities

3 Likes
115

Just thinking loud here, canā€™t we just conditionally wrap the main function based on the presence of a ā€œFiberā€ package or some config? that way the ā€œsomehow refactored core packagesā€ can run without Fiber but if dev have legacy code or want to use Fiber they can add that package.

Same here.

1 Like
116

Right, if there is a way to be backwards-compatible. The point is, we need something for server-side methods for async. Fibers is one (good) way, we can simulate other ways to have more appeal.

117

Just to throw in my idea of backwards compatibility in this picture - using mongo as an example:

  1. Take the package which currently use Fibers, refactor it to use promises and rename it to mongo-promise.
  2. Create a new package, depending on mongo-promise, which waits for the result of the promise by using Fiber - and thereby establishes the old behavior. Call it mongo-fiber.
  3. Create a new package having the same name as the package initially had (mongo) and let it only depend on mongo-fiber.

This will give the core-team the possibility to e.g. add a deprecation-notice in a later version and give people the choice to either switch to mongo-fiber if they want to keep fiber, but encourage them to switch to mongo-promise - or to leave it in place.

In this case:

  1. People can themselves decide when to switch
  2. It can run side-by-side for an undefined while until we decided definitively where to go
  3. Even if the core components wonā€™t switch, the individual developer will have the possibility to use promises.

On the other side, this might create a big confusion when dealing with older code (e.g. if a package never would release a new version but remain at a fiber-based solution but other plugins will - and you then will have to deal with the mess that half of your mongo-instances now are async while others use fibers).

Just my $0.02 ā€¦

7 Likes
118

I like your approach. I understand your worries, It all boils down to strategy and tactics. If the strategic decision has been made to switch to async, it should be put down everywhere. New pull requests or packages that rely on Fibers would for example not make it into the core from that point on. Over time, the dept has to be solved.

1 Like
119

I think this thread has gone long enough. I think if you want to discuss the specifics create a new thread and link it here. I would ask you to focus here on any other conceptual ideas (if we havenā€™t exhausted this topic already :smiley: ).

:peace_symbol:

1 Like
120

Something sad that I used to like in Meteor was the WOW! NO IMPORTS!!!

But later they introduced the new import build.

As long as you donā€™t want to precisely call file order, why not import everything directly?

And what if I want to control import order? A numbered list is created in a .meteor file after each meteor command, and can be edited manually for the next build to occur in the correct js file order.

The idea is that the webdesigner can easily control the load order with as minimum boilerplate as possible.

Then all my componentā€™s code will be simpler.

1 Like