Thingking about Meteor 2.0 šŸ¤”

core
108

@gaurav7

Letā€™s come to the reasons we use Meteor.wrapAsync (or Future like we do directly).

First, in a server-side method, you have to tell the framework how it can send the result back to the user (i.e. the client). An asyncCall by itself is not enough as your callback function is not in the main thread of the server-side method. I suspect other frameworks use something similar.

The second reason which is within the code for Meteor.wrapAsync you will notice that the callback is wrapperd in Meteor.bindEnvironment ā€“ it binds the callback to the current environment. You need this if within your callback you are making DB changes (otherwise itā€™s not needed).

You canā€™t get around either of these in ANY framework. So the question is: How can we make it transparent for non-Meteor users to lessen the learning curve (and adoption barrier).

The trend right now is using Promises. Or, if there is a way within the Meteor compiler to use async/await within the server-side method, then great. I donā€™t know enough about JS compilers to make a statement.

So to answer the question, I would opt for an internal implementation of Promise to get around this (a bit like the AWS SDK as mentioned above)

109

@ramez I do understand why Meteor.wrapAsync and Future (and similarly Meteor.bindEnvironment and meteor/promise) are used in Meteor today. This thread has become quite long and maybe you didnā€™t get a chance to read my earlier comments, but I appreciate why these decisions were taken by Meteor. The crux of my argument is only that the time has come now for Meteor to revisit that decision, keeping in mind the current state of Node.js and its ecosystem ā€“ just like supporting NPM packages was a necessity to keep Meteor relevant within the Node.js landscape, and it materialized in v1.3.

Btw, if I understood these points correctlyā€¦

ā€¦it seems youā€™re not considering that Meteor already supports returning a promise from server-side method? E.g.

Meteor.methods({
  uploadToS3() {
    return s3.upload({}).promise();
  },

  async uploadToS3AndLog() {
    const response = await s3.upload({}).promise();
    console.info('s3 upload response:', response);
    return response;
  }
});

ā€¦ both methods return the response to the client-side, after calling the s3 api asynchronously.

If more and more external libraries support promises, maybe Meteor developers would need to use less and less of fibers-specific helpers.

But App-Developer Experience (that the above example relates to) is not the only reason to remove fibers, nor the most important one. There are multiple reasons why removing fibers seems like a must-do to me, even if it calls for a backward-incompatible v2.0. Let me rank them here by the quantum of positive impact:

  1. Meteor would not only get rid of all known fibers related bugs and incompatibilities (see: comment-44 and comment-90), but also the unknowns (nobody knows how many of them are waiting to be discovered), and also the ones that might get introduced in the future (such has been the unfortunate history of Meteor with fibers).
  2. Not having to maintain a non-native paradigm of asynchronous programming would free up the already few maintainers to focus on things that matter.
  3. The absence of this alien concept (alien to the majority of Node.js developers) would encourage more developers to adopt Meteor, and consequently more library authors and tool developers.
7 Likes
110

I had a quick look at how everything is bundled. Apparently boot.js runs the individual packages within a Fiber


ā€¦ and I guess itā€™s up to the individual APIs to call Fiber.yield (or any other method that does something similar).

So I believe it might be possible to refactor each package and tell the bundler to not run those packages inside a Fiber. If this path could be taken, then maybe this exercise would not need a big-bang change and it might also be easier for the community to contribute. But I canā€™t say for sure, since I am not aware of the implicit coupling between the packages and the bundler. Also not sure if this can be done in a backward-compatible way.

Though I think Iā€™m satisfied enough to at least propose this on Github and then leave it to the wisdom of Ben and co; shall do so soon.

3 Likes
111

Thatā€™s not quite true. The Promise is resolved in the method and the (normal) result is returned. As it happens, Meteor uses Fibers to achieve this behaviour.

112

@robfallows I meant the same :slight_smile:.

Not sure how I miscommunicated to you, perhaps I should have also shown that the following works:

if (Meteor.isClient) {
  Meteor.call('uploadToS3AndLog', (uploadResponse, err) => {
    // `uploadResponse` matches `response = await s3.upload({}).promise()` on server-side
  })
}

It appeared to me that @ramez seems to think using Future is the only way to return the result of server-side async operations to client-side, to which I clarified that my example would also work if the server-side async operation is done via promises.

Update: @robfallows ok I see now that what I wrote could be interpreted as the client also receiving a Promise. Like you rightly said, thatā€™s not how if works. The client-side handling remains the same. My focus was on being able to return the response of an async server-side operation using promise

2 Likes
113

You are not sure because I did not state any support or non-supportā€¦ :wink: My intent was to get people to think.

If you are talking about removing Fibers in the next release of Meteor, as someone with a few apps deployed I will not look at it with excitement, due to all the re-work needed if I need to upgrade. The paradigm of asynchronous code is not a natural fit to developers transitioning from pre-Node.js server side platforms and this applies to most corporate/enterprise developers. Server-side Meteor is really easy on these developers, specifically because the async nature on Node is abstracted for them, specially for MongoDB work. I do understand the appeal to people coming from Node.js thoughā€¦

So, I sit on the fenceā€¦ Maybe leaning towards not breaking all the apps out there while Fibers is still usable.

A few old discussions on the subject for reference:

3 Likes
114

Cool, my intention behind saying so was to make you reveal your intent :smiley:

Thanks for sharing the previous discussions on the topic, I should have searched the repo issues harder. Gonna go through them to see if itā€™s worth re-igniting the discussion in the repo.

Btw, Iā€™m not at all suggesting fibers should be removed in the next release. The OP asked about Meteor 2.0 and I thought that would be the best time to revisit a fundamental decision like fibers considering todayā€™s realities

3 Likes
115

Just thinking loud here, canā€™t we just conditionally wrap the main function based on the presence of a ā€œFiberā€ package or some config? that way the ā€œsomehow refactored core packagesā€ can run without Fiber but if dev have legacy code or want to use Fiber they can add that package.

Same here.

1 Like
116

Right, if there is a way to be backwards-compatible. The point is, we need something for server-side methods for async. Fibers is one (good) way, we can simulate other ways to have more appeal.

117

Just to throw in my idea of backwards compatibility in this picture - using mongo as an example:

  1. Take the package which currently use Fibers, refactor it to use promises and rename it to mongo-promise.
  2. Create a new package, depending on mongo-promise, which waits for the result of the promise by using Fiber - and thereby establishes the old behavior. Call it mongo-fiber.
  3. Create a new package having the same name as the package initially had (mongo) and let it only depend on mongo-fiber.

This will give the core-team the possibility to e.g. add a deprecation-notice in a later version and give people the choice to either switch to mongo-fiber if they want to keep fiber, but encourage them to switch to mongo-promise - or to leave it in place.

In this case:

  1. People can themselves decide when to switch
  2. It can run side-by-side for an undefined while until we decided definitively where to go
  3. Even if the core components wonā€™t switch, the individual developer will have the possibility to use promises.

On the other side, this might create a big confusion when dealing with older code (e.g. if a package never would release a new version but remain at a fiber-based solution but other plugins will - and you then will have to deal with the mess that half of your mongo-instances now are async while others use fibers).

Just my $0.02 ā€¦

7 Likes
118

I like your approach. I understand your worries, It all boils down to strategy and tactics. If the strategic decision has been made to switch to async, it should be put down everywhere. New pull requests or packages that rely on Fibers would for example not make it into the core from that point on. Over time, the dept has to be solved.

1 Like
119

I think this thread has gone long enough. I think if you want to discuss the specifics create a new thread and link it here. I would ask you to focus here on any other conceptual ideas (if we havenā€™t exhausted this topic already :smiley: ).

:peace_symbol:

1 Like
120

Something sad that I used to like in Meteor was the WOW! NO IMPORTS!!!

But later they introduced the new import build.

As long as you donā€™t want to precisely call file order, why not import everything directly?

And what if I want to control import order? A numbered list is created in a .meteor file after each meteor command, and can be edited manually for the next build to occur in the correct js file order.

The idea is that the webdesigner can easily control the load order with as minimum boilerplate as possible.

Then all my componentā€™s code will be simpler.

1 Like
121

I had an idea some days ago and might be in line with the lets say marketing strategy of meteor.

Why not create a meteor-community github organization and rebrand some of the most important packages that have been abandoned and or their mainteiners agree to change the ownership.

I think this could have a possitive effect having a more up to date and better maintained repos that can make improve the confidence for newcomers.

What do you guys think about it?

2 Likes
123

Well, the old system without imports still works. But having everything available globally easily becomes unmanageable as your application grows. And more important, it makes Meteor in line with the javascript ecosystem. Meteor started out with its own packaging, module system, code-style, etcā€¦ While many of these decisions were well motivated and often way ahead of their time, they made Meteor stand alone on an island. And as the JS ecosystem evolves it became hard to keep up and developers left Meteor.

5 Likes
124

I have my concerns about this because youā€™re basically giving plugins then on the hand of many where not everyone is cooperating with everyone else. You can easily have people who ā€œjust want to fix something smallā€ and start pushing out a new version as they please etc.

Maybe itā€™s a part of my german culture, but I like if someone is responsible for things. This is also why (in my eyes) packages should have maintainers and not just be ā€œat the communityā€.

Iā€™d rather personally pick up packages which have been abandoned and ask the author to get access to github and atmosphere to publish new versions on their behalf - or join groups having packages which are related to each other (like the meteor-testing group).

Iā€™ve also started forking packages and pushed out a notification in this forum about it and in the ticket system in case the author was not responding to my request (either approving or denying - but just keeping silent) for weeks or sometimes even months.

2 Likes
125

I know I will lose the battle in advance because you all chose the ā€˜community aspectā€™ first and my opinion is unpopular.

I will share it still. Maybe some people will stop being dogmatic and actually listen :crossed_fingers:

I argue that there should be the choice: all fine for current imports system. Letā€™s keep it.

The concept of automatic imports should be pushed further.
This is so handy for small teams, and for prototyping ! Which is where Meteor (used to?) shine

The automatic import system is still working, but now it seems to be covered with dust and didnā€™t get proper attention recently.

Having a system to fine-tune load order easily as I suggest would dramatically simplify the codebase for small projects.

Being able to control load order means that the most important scripts will be loaded first and I can control that easily as well.

I just thought this was a nice option, especially for beginners. God it seemed so easy to get started on Meteor before, but since imports are a part of the Meteor tutorial, I see a lot more complexity from a beginnerā€™s point of view.

2 Likes
126

No worries. When systems scale, it becomes hard to chose what goes in and what not. For this reason its good to look at what users want. It probably seemed like a good idea to go import based, because the auto import version had issues regarding scale.

Unfortunatly this decision also impacted the entry level for new devs and prototypers

1 Like
127

You can already fine-tune the load order by naming convention thanks to the rules of automatic loading here: https://guide.meteor.com/structure.html#load-order
ie. /lib/ first, then depth first search, each folder ordered by filename.

As soon as you create a file with a list of imports in order, you might as well use native importsā€¦

1 Like
128

You can already fine-tune the load order by naming convention thanks to the rules of automatic loading here: https://guide.meteor.com/structure.html#load-order

I have a huge problem with that, since the file structure is the number one thing that makes a codebase understandable and clear. Iā€™ve been dealing with having to prepend my filenames with numbers, just because I needed to gain control over the load order. Alternatively Iā€™ve dealth with writing atmosphere packages that wrapped my app code just because in those packages I did have control over the load order.

I want to ppint out another thing. Learning curves are not a bad thing. They need to exist in order to make developers better. Not using imports and all of its benefits removes part of the required skills to properly structure applications and opens the next curve towards learning about NPM and dynamic imports / code splitting

4 Likes