Thingking about Meteor 2.0 🤔

[storyteller]

Well, now comes the hard part. Actually making it happen…

[vlasky]

No, I don’t support killing DDP and Minimongo as these are core parts of what makes Meteor Meteor and provide functionality that many long term users (like me) rely on in our Webapps.

Providing the option to use/not use DDP and Minimongo or to programmatically turn them on and off is fine.

As for Fibers, I am not sure what the issue is. Promise.await() is easy to use and works perfectly when calling promise-enabled functions or those that use async/await

Ben Newman gave an excellent talk in 2015 to explain why Meteor should retain the use of Fibers.

Slides:

http://benjamn.github.io/goto2015-talk/#/

I sincerely ask - what has changed since Ben’s talk that would make a complete migration to async/await worth doing?

Are there any issues with Fibers in Node.js 10?

[cloudspider]

I totally agree on the Minimongo and DDP part. Its for me one of Meteor’ biggest strengths. Not sure about Fibers though. Its on one hand confusing for developers coming from other platforms. On the other its a very convinient way of writing.

[alawi]

of what makes Meteor Meteor

As I said before here, the concept Meteor means different thing to different people, so we’ve to keep the discussion in the perspective/context of how the tool is being used.

As for Fibers, I am not sure what the issue is.

No, I don’t support killing DDP and Minimongo as these are core parts of what makes Meteor Meteor and provide functionality that many long term users (like me) rely on in our Webapps.

Providing the option to use/not use DDP and Minimongo or to programmatically turn them on and off is fine.

I agree, I use Minimongo and DDP myself but that was @sacha statement from here. And I think they didn’t mean to kill minimongo/DDP but to make it an optional package since those using GraphQL don’t really need the web sockets open which consume server memory and has slight initiation delay at the client, so making it optional, in my opinion, is a reasonable ask since they’re using Meteor just as a backend for a GraphQL end point.

As for Fibers, I am not sure what the issue is.

I’m not sure either but folks who advocating for it are saying it’s a barrier to move Meteor to NPM, it’s holding some developers coming from the rest of the NodeJS ecosystem off guard, and they might also be barrier for contributing to Meteor core, those points have been echoed several times in this thread but I think that requirement could be flushed out more because, for me, it still sounds vague but maybe it’s my lack of experience with Meteor core. You can see @gaurav7 comment here when I questioned why Fiber is an issue. Also, Ben did highlight some of the issues that comes with using Fibers, especially catch 3 in his slides. I think in his last comment you can see that he favoured the tradeoff of simplifying Meteor usage (developers experience) on the expense of the effort it takes Meteor core contributors to learn and maintain Fiber (contributors experience).

[alawi]

Well, now comes the hard part. Actually making it happen…

Well I think you started a good thread, I think it’s worth outlining and capturing the community needs/pain points/sentiments. But yeah execution is another story, I’m seeing progress on the Vue guide, which I think is a real service to Meteor that the community is well positioned to independently help with.

[gaurav7]

As for Fibers, I am not sure what the issue is. Promise.await() is easy to use and works perfectly when calling promise-enabled functions or those that use async/await

Ben Newman gave an excellent talk in 2015 to explain why Meteor should retain the user of Fibers.

@vlasky see my earlier comment highlighting the parts of Ben’s presentation where he seems to acknowledge the points being echoed in this thread w.r.t Fibers.

Sure Fiber makes many things appear “easy”, as long as it works. But it’s a nightmare when it does cause a problem, the likelihood of which increases as you scale and/or use more 3rd party libraries. So far, the “ease of use” argument perhaps rightly trumped over the cons, justifying the tradeoffs. But there was no alternative then. Now we have one, in the form of async/await, and the best part is that it’s a proper standard!

Not only would removing Fiber make Meteor more approachable to other Node.js app developers and authors of libraries & tools, but it would also save the few precious resources currently working on the framework (mainly Ben, as per github stats) from spending time on unproductive work (i.e. no new framework feature) that Fiber keeps demanding… just look at some of the current open issues to get an idea:

• https://github.com/meteor/promise/issues/14
• https://github.com/meteor/promise/issues/9
• https://github.com/meteor/meteor/issues/10117
• https://github.com/meteor/meteor/issues/10279
• https://github.com/meteor/meteor/issues/9275
• https://github.com/meteor/meteor/issues/10032

All this just to pretend that we’re writing synchronous code on a platform that’s inherently asynchronous! Is that really worth all the trouble now that there’s a good/better alternative?

As an example of the overhead it would eliminate: the whole meteor/promise package won’t be needed if Fiber were to be removed! Note that Promise.await and Promise.async are not standard APIs.

[cloudspider]

agree, I use Minimongo and DDP myself but that was @sacha statement from here. And I think they didn’t mean to kill minimongo/DDP but to make it an optional package since those using GraphQL don’t really need the web sockets open which consume server memory and has slight initiation delay at the client, so making it optional since and using Meteor just as a backend

Yes exactly this is why I’m now supporting simpleDDP and am building my vue connectors for it.

It feels to me that the red line here is pluggability. It takes quite an effort just to understand Meteor’s internals. Making this easier and putting developers in control over what functionality they require is crucial. That’s one thing I love about Vue and Nuxt. The plugins

[alawi]

Do you think Fibers can be removed in a backward compatible way (i.e legacy app code still works as is)? It sounds to me that removing Fiber from the Meteor core is going to be a major refactoring given the potential impact/testing required.

But it’s a nightmare when it does cause a problem, the likelihood of which increases as you scale and/or use more 3rd party libraries

I’ve personally never experienced an issue with underlying cause pertaining to Fiber usage and we’ve tons of third-party npm packages so I can’t relate to that statement, and being on this forum for sometime, I don’t see many issues pertaining to Fibers usage either. Also it seems coroutines are emulating multi-threading and multi-threading bugs are nightmare in general. However, there are some hard corner cases issues reported on Github pertaining to Fiber use such as this one…

All this just to pretend that we’re writing synchronous code on a platform that’s inherently asynchronous!

Human mind is naturally sequential thus having Meteor backend synchronous by default is in an advantage and one less hurdles for newcomers before getting their todo list render on the screen.

Is that really worth all the trouble now that there’s a good/better alternative

Actually I’ve the same question toward removing Fiber, is that really worth the trouble? If I understood Ben’s talk correctly, coroutines (Fibers) is a more generic/flexible approach toward async code when compared to async/await and he demonstrated how async/await can be implemented using coroutines furthermore it does enhance the developers experience (in Java we’ve synchronized blocked, other languages (Kotlin for example) have co-routines as well as Ben’s alluded too). There is a great talk from Kotlin authors explaining why coroutines are superior to async/await, they made very similar points to Ben, you can find it here. So my personal take on the refactoring of Meteor Fibers, that it’ll be a major undertaking (which I think an effort could be spend elsewhere) that doesn’t improve the developers experience (in fact it might degrade it) and I really don’t think it’s the main hurdle for contributing toward Meteor (I think it’s more marketing issue) so for me the refactoring is not justified, but again I keep an open mind, I think it’s debatable topic.

[ramez]

@alawi, agreed.

Maybe we need to be public with a way to abstract it out for NodeJS developers thinking of using Meteor, via stubs for Promise. If I look at some of our method calls, we do use Fibers a lot for async calls (e.g. communicating with AWS, files etc.) so I can see why developers would currently have no choice but to interact with Fibers.

But refactoring Meteor to extract Fibers would be writing a whole new framework.

[alawi]

If I look at some of our method calls, we do use Fibers a lot for async calls (e.g. communicating with AWS, files etc.)

Sorry maybe a little off topic, but I’m a bit confused on why using Fibers for async calls on the server? I’m a bit puzzled why I didn’t have to interact with Fibers, I do make a lot of async calls to third party APIs/filesystem but I just use promises/async await or Meteor.wrapAsync, so when exactly do I need to use Fiber, am I missing something, why I haven’t experienced the “Fiber pain” yet ?

[ramez]

When you make a call to a Meteor method, and that method depends on an async call (e.g. pushing a file on S3) how else would you do it?

We return a future.wait() in the Meteor method, and in the callback from S3 we return future.return(result) or future.throw(error)

How else would would you do it?

EDIT: Fixed text to replace fiber with future

[ramez]

I just checked Meteor.wrapAsync (which we don’t use – maybe we should change that). It takes care of that for you. It also does the Meteor.bindEnvironment for the callback.

Maybe this is the solution. Maybe Meteor.wrapAsync could be the starting point of using Promise (developers don’t care what’s under the hood)

[alawi]

Yes, I used Meteor.wrapAsync for the Stripe calls convert the async calls to sync and I wait for the results, the code block reads sequentially thus making it easy to reason about, that’s why I said removing Fiber might actually degrade the developer experience. For file uploads, I push to S3 from the client. In both cases, I wasn’t even aware of Fiber when working on them!

By the way I highly recommend Rob Fallows article here, really helped me to wrap my head around Meteor’s async.

[gaurav7]

But refactoring Meteor to extract Fibers would be writing a whole new framework.

Yeah very likely it would be a rewrite at least for the parts that are tightly coupled with fibers. That’s why I brought this up only in the context of a hypothetical Meteor 2.0 (which I wish doesn’t stay hypothetical for too long ).

[gaurav7]

Maybe this is the solution. Maybe Meteor.wrapAsync could be the starting point of using Promise (developers don’t care what’s under the hood)

The latest versions of AWS SDK have .promise() methods. So you could also do something like

async function upload() {
   const response = await s3.upload({}).promise()
}

or pretending synchronous code through implicit use of fibers:

function upload() {
   const response = Promise.await(s3.upload({}).promise());
}

So I must clarify in my push for removal of fibers, that it’s entirely possible to write async/await code in Meteor today. And with the non-standard helpers like Promise.await and Promise.async you are able to interact with external promise-based APIs.

[gaurav7]

that’s why I said removing Fiber might actually degrade the developer experience

// node_modules/external/index.js:
export function asyncCall(callback) {
  callback(err, result);
}

// non-Meteor Node.js
import { asyncCall } from 'external';
asyncCall((err, result) => {})

// Meteor Node.js
import { asyncCall } from 'external';
Meteor.wrapAsync(asyncCall, this)((err, result) => {})

Which one has a better Developer Experience?

To be fair, DX is also a function of what you’re already used to. When Meteor was launched, Node.js was also fairly new, there was no (popular) Rails-eque fullstack web-framework, callback hell was one of the early criticisms of Node.js. So giving the appearance of synchronous code might have been a masterstroke then to attract developers from other platforms like PHP and Java.

But today, Node.js has arguably surpassed the popularity of the traditional platforms. Co-routines are not idiomatic to Node.js for whatever reason. So if Meteor wants to go back to “mainstream” and lower the barrier to entry, I believe it should eliminate aspects that are not familiar to Node.js developers trying out Meteor.

So the use of fibers not only enforces technical tradeoffs that are not worth anymore, I believe it has also made Meteor framework “not worth the extra learning curve at a fundamental level”, as modern alternatives keep coming up.

[gaurav7]

Do you think Fibers can be removed in a backward compatible way (i.e legacy app code still works as is)? It sounds to me that removing Fiber from the Meteor core is going to be a major refactoring given the potential impact/testing required.

I’m not familiar with Meteor internals, but prima facie I expect it to be quite a major work, almost a rewrite of all code that’s coupled with fibers. That’s why I made the suggestion only in the context of Meteor 2.0.

Every API that hides the asynchronous execution would have to be changed to be Promise-based. So

const user = Meteor.users.findOne();

might become

(async () => {
  const user = await Meteor.users.findOne();
})(); // framework may support top level `await` but let's skip the nitty-gritties iin this discussion

or its equivalent:

Meteor.users.findOne().then(user => {})

Hopefully there would be a way to make the current fibers approach an opt-in, maybe via a package. Somebody familiar with how fibers are used internally might be able to tell how backward compatible that could be made. But if that’s not feasible, it shouldn’t stop this evolution; then Meteor 2.0 could be a recommendation for new apps while current apps could stay with 1.x and gradually migrate if worth it for that individual project.

I don’t see many issues pertaining to Fibers usage either

Well, “many” might be subjective, but I think there are enough to warrant a revisit of the decision. More importantly it’s the nature of those issues. And perhaps even more importantly, any such problem is doomed to be specific to Meteor; the rest of the Node.js community is unlikely to find any incentive in fixing them. See the example I shared earlier.

Human mind is naturally sequential

Today, this problem can be considered “solved” in Node.js because of async/await. Since Meteor is not a language, but rather a framework on Node.js, it would be better to defer to Javascript/Node.js standards where feasible – if we were to apply the principle of least astonishment here. A newcomer is more likely to read about Node.js first than Meteor.js, right?

Actually I’ve the same question toward removing Fiber, is that really worth the trouble?

Sure that’s the decision to be made, and I hope the points I’m putting forth would help with that. I have come to conclude that while adopting fibers was likely a good decision at the time, it has become a technical debt considering the current state of Node.js and its ecosystem, and of course also considering the current state of Meteor.js.

If I understood Ben’s talk correctly, coroutines (Fibers) is a more generic/flexible approach toward async code when compared to async/await and he demonstrated how async/await can be implemented using coroutines

I don’t think he said coroutines is “better” than async/await, just that with coroutines we won’t need those separate keyword. But forward to today, we have those separate keywords as part of the language itself! And coroutines may never become a standard in JS (for good reasons).

I’ve explained my take from his presentation earlier; pasted below is the part that addresses the specific points you raise here:

My interpretation of the presentation is that Fibers helped Meteor to adopt async/await in a backward compatible way. But I don’t think Benjamin made any arguments about Fibers being worth sticking with even in future. In fact he seems to support the idea that async/await is the solution (slides: 20 , 21) for coroutines not being suitable for Javascript (slides: 18, 19)

(in Java we’ve synchronized blocked, other languages (Kotlin for example) have co-routines as well as Ben’s alluded too). There is a great talk from Kotlin authors explaining why coroutines are superior to async/await, they made very similar points to Ben, you can find it here.

My point is not whether coroutines are “better” as a concept. My simple submission is: is Meteor still compelled to live with the tradeoffs of using fibers? If Meteor were to be written from the ground up today, would one have still used fibers? I reckon that removing fibers would not only eliminate the technical debt whose interest has to be paid in the form of the kind of issues I listed earlier, but also help with the adoption and community contribution.

[alawi]

Yes, but I only had to use wrapAsync for third party API the rest of our code is all sync which is easier to read. So 95% of our code read sequentially (all the DB calls) thanks to Fiber and the 5% (even less in our case) when we call third party API we just wrap it in a function.

A newcomer is more likely to read about Node.js first than Meteor.js, right?

Well it depends, I came from Java and the sequential nature of Meteor appealed to me, I agree with the folks from Kotlin, I do think it’s the better and more natural way to handle async.

I understand you point/concern @gaurav7 that Meteor is diverging from how async is now being handled in a typical modern node backends and yes we might end up in some niche issues, issue (#9796) is the most interesting, which turned out to be V8 defect on how threads being managed. But I think given the history, available resources, and even tradeoffs (there are strong arguments for using coroutines to handle async), I personally think the effort/discussion for Meteor 2.0 should be focused somewhere else but anyway I think that’s the core maintainers call, since they’re impacted the most by this refactor, so I rest my case

[gaurav7]

I personally think the effort/discussion for Meteor 2.0 should be focused somewhere else but anyway I think that’s the core maintainers call, since they’re impacted the most by this refactor,

Agreed. I think the biggest impact of removal of fibers would be to make the maintainers’ lives easier once the feat is accomplished; then they won’t have to deal with all the overhead

[cloudspider]

We should at least stop depending on Fibers when it comes to the new packages. Maybe over time we can refactor some of our most valuable packages to become async / await. For me it might be cool to discover just exactly how Minimongo works while adding a new feature hydration.