Fibers Public Roadmap and Meteor 3.0

denyhs · November 6, 2023, 7:59pm

As for the current version, you have both. So you can keep on using Accounts.createUser.

Accounts.onCreateUser does not need an async version, as it just receives a callback to be called whenever a new user is created.

denyhs · November 6, 2023, 8:11pm

Hello everyone! Here is another update for you!

In the past few weeks, we addressed issues like this, and we have a new Alpha, 3.0-alpha.17.

We now have a new document where you can see the main tasks we need to finish until we have a final release for version 3.0. You can check that out here.

Also, for the next two weeks, the main tasks we’ll be addressing are Unhandled Rejection when fails to bundle the app and Review Meteor.call, Meteor.callAsync, Meteor.apply, Meteor.applyAsync so we can release a final Alpha version.

See you soon for more updates!

charlytalavera · November 7, 2023, 3:33am

Thanks for your reply @denyhs! That sounds great, we’ll start migrating in the meantime, thanks.

charlytalavera · December 11, 2023, 12:05pm

Hi! Is there any update? Do you think that we will have a Beta release for this month? I was checking the backlog already but looks like there are a few tasks pending yet for the beta release. Not really sure if it’s just outdated or if the Beta release will be delayed.

Thanks!

denyhs · December 11, 2023, 8:30pm

Hi, we’re finishing things with our Meteor.callAsync API and that was the main thing holding us back.

Hopefully, this week, we’ll have this finished, and we can move on with the beta.

denyhs · December 13, 2023, 4:35pm

Hello everyone! Here is a quick update for you on what we’ve been doing in the past few days!

Our Tracker.autorun now has a new attribute called firstRunPromise, and now you can await for the invoice of this method. You can find the PR here, and more details about how to use it in this new blog post.

We’ve also been working with Meteor.callAsync / Meteor.applyAsync API. We needed to make the API easier to use. It wasn’t trivial work, but we’re basically finished with it. The work is divided into two PRs. You can check the first one here and the second one here.

HUGE thanks to our community on this one. We had a lot of feedback and participation on it. And special thanks to @zodern, who created this code. You can check their post about it here.

We also started working on our Migration guide and updating our Docs for version 3.

Unfortunately, we had some drawbacks in the past couple of weeks, but we’re still confident that we can have a Beta version by the end of the year.

So stay tuned, and see you soon for more updates!

You can follow the updates on our overview doc, Fibers Public Roadmap and on this Github PR .

seba · December 27, 2023, 10:02am

I’ve been out of loop with Meteor for a while, but I see that Tracker now support async functions. But obviously this won’t work with tracker dependencies since the current computation context is lost. What’s the plan to make this work, since this seems quite core to Meteor?

grubba · December 27, 2023, 2:40pm

Hey!! It now works! We found a way to make it reactive. It does require a little bit of extra work, but you can maintain the reactive context using Tracker.withComputation function more can be seen in the docs

DanielDornhardt · December 29, 2023, 5:53pm

Also, as a stopgap measure, if you have more complex nested reactive code in the client, you can use our babel plugin to wrap code with Tracker.withComputation automatically and keep the reactive context across async/await calls:

trusktr · January 7, 2024, 8:37pm

Unless I there’s existing conversation I’ve missed, it seems that the value of native ES modules are being overlooked in the Meteor 3.0 work, and I think it would be crucial to expose a pathway that allows folks to use fully-native JS modules in Meteor 3.0 (for example opting out of ecmascript and modules packages).

I updated the todo list discussion with more details, including links to what I’ve already tried, in this comment: What's left until an official Meteor 3.0? · meteor/meteor · Discussion #12865 · GitHub.

It seems fairly simple, without changing or removing existing ecmascript stuff, to at least enable a pathway to native ES modules.

If you do this, then people will be able to flexibly attach their own build tools to their Meteor workflow without being locked into a system that is already legacy.

Then, a future Meteor 4.0 could delete ecmascript (etc) and provide a better alternative, while at least people on Meteor 3.0 would be able to bypass Meteor’s file handling and simply manage their source code using any of today’s standard tools (or no tools at all, just plain modules).

It seems like the missing piece is an option that enabled import() inside of Node vm instances. Once we have that, we’ll be able to import native ES modules in Meteor (see the links in the comment).

denyhs · January 8, 2024, 3:53pm

I’ll have a look at this. Maybe get some help from @zodern. I think they would have really good insights on this.

denyhs · January 11, 2024, 9:13pm

Hello everyone! Here is a quick update on what has happened in the last month!

We launched the first Beta version !

You can test by running: meteor update --release 3.0-beta.0

Meaning that we finished the work with our Meteor.callAsync / Meteor.applyAsync API.

Something worth mentioning again is that this version is already running with Node 20!

Also, we’re working hard to finish migrating and updating our docs and guides. You’ll hear more about it soon.

We’re now moving forward with the Beta, and your help with tests and feedback will be of great value!

So stay tuned, and see you soon for more updates!

You can follow the updates on our overview doc , Fibers Public Roadmap and on this Github PR

vblagomir · January 11, 2024, 10:32pm

Thank you guys, you rock! I can only imagine monumental work behind this. Will check as well, it is about time.

zodern · January 12, 2024, 8:42pm

If we limit the scope to only app code on the server, it might not be too much work to add an experimental mode for native ESM. This doesn’t have to be part of Meteor 3. If we make it optional, we can add it in 3.x and iterate on it. There would be many breaking changes with native ESM.

Some of the things involved would be:

updating import scanner to handle ESM
not linking or having a simplified linking process
changing the boot process
making sure source maps on the server still work correctly
the current implementation of Assets requires there to be an app scope, which there wouldn’t be with ESM
allow app code to import packages in a way that live bindings still work
figure out how to handle packages and apps importing the same npm package to avoid having multiple instances of it where Node wouldn’t
change how core-runtime detects when all code is loaded
add a resolve hook so Node can find the Meteor packages and the app’s npm dependencies

The more interesting part is the client. For small apps, using native ESM might work, but the loading time scales with the number of modules, so at some point it becomes too slow. There’s also a number of features missing in ESM that Meteor would be expected or required to have. I’ve been thinking about this for a while. When I find the time, I will write a post describing the details, different options, trade-offs, what other bundlers have done, etc.

Three of the goals we would definitely have are:

reasonably fast loading time even for apps with 10,000’s of modules
Consistent behavior with the server’s module system
Support for npm packages that use commonjs

I don’t think Meteor’s current module system is that outdated. If we designed a new module system today, it might look very different, or, depending on the goals, features, and the trade offs you were willing to make, you might end up with a very similar design. Some new projects, such as Turbopack or bun, have many similarities with Meteor’s module system. Having said that, I am very interested in alternative designs that have the same benefits of the current system and minimize the issues with it (for example: bundle size, build time complexity, runtime complexity, some core packages unable to use modules). Since we would need the client and server module systems to behave consistently, while experimenting with native ESM would be a good time to explore this.

Before investing in native ESM, it might be good to fix the two main issues with Meteor’s current module system:

there is one code path left in reify that we never optimized, and in some cases can add a few seconds to the page load time
update Meteor’s resolver to support the package.json exports field. This probably will be needed even if we switch to native ESM

trusktr · January 13, 2024, 12:24am

I think breaking changes can be avoided. All Meteor APIs are already exposed globally via Package.*. By implementing an ESM customization hook, the Package.* references can simply be exported to the import statements of any native modules.

This will be very similar to what I’ve done with native client-side ESM: my import map simply maps the specifiers to a file that exports stuff from Package.*. (Note, import maps are on the Node.js roadmap as an alternative standards-aligned way to achieve this later).

Customization hooks also allow transpiling content (f.e. they can read a .css file (from the specifier of an import statement), and make the default export be a string containing that CSS, and we can do similar with TypeScript, etc).

File caching would be needed to avoid compilation every single restart if files haven’t changed. A customization hook could also communicate with a dev server that provides cached results.

Here’s the Node’s example on how to begin to add URL imports like Deno: Modules: node:module API | Node.js v21.5.0 Documentation

Here’s a sample of what implementing meteor/foo modules might look like:

// meteor-hooks.mjs
export function load(url, context, nextLoad) {
  // Handle 'meteor/*' specifiers
  if (url.startsWith('meteor/')) {
    const lib = url.split('/')[1] // f.e. "tracker" in "meteor/tracker"
    return { format: 'module', source: `export default Package['${lib}']` }
  }

  // Let Node.js handle all other URLs.
  return nextLoad(url);
}

User code for that implementation pattern would need to get the default instead of a named export, and type defs would need to be adjusted, but it would work:

import tracker from 'meteor/tracker'
const {Tracker} = tracker

Alternatively, if we ensure that meteor packages all export the Uppercase name of the package, we can make it a named export:

// meteor-hooks.mjs
export function load(url, context, nextLoad) {
  // Handle 'meteor/*' specifiers
  if (url.startsWith('meteor/')) {
    const lib = url.split('/')[1] // f.e. "tracker" in "meteor/tracker"
    const pascalCased = dashToPascalCase(lib) // f.e. tracker to Tracker or foo-bar to FooBar
    return { format: 'module', source: `export const ${pascalCased} = Package['${lib}']['${pascalCased}']` }
  }

  // Let Node.js handle all other URLs.
  return nextLoad(url);
}

And then user code would be more like today’s:

import {Tracker} from 'meteor/tracker'

Doing it on native ESM with customization hooks (and/or with import maps later) you won’t have to worry about this. The native module resolution algo will handle that.

No longer resolve hooks (CommonJS), but customization hooks.

That’s a good point. I’ve yet to experience a problem big enough where I needed to bundle my app (with native CommonJS or with native ESM), but it imagine it could certainly be needed.

Bundling would be an optimization for that case, but I think it should be built anew, not based on the ecmascript package, but perhaps on something like esbuild or Vite if not something custom. It will need to support standards like import.meta.url which, when bundled, should not change meaning (console.log(import.meta.url) should log the same both with no bundling or with bundling).

I think the first step will be to get the native ESM mode working, without any such optimizations up front.

Trying to do it all at once will be far too much work, and having a plain ESM mode will already be ideal to begin with: people can start to use their own build tools to target native ESM mode, f.e. compile their own TS to JS, while existing users with existing apps can, for now, continue to use ecmascript and typescript packages.

An initial native ESM mode will usable for new Meteor apps, while existing apps may have migration work to do, but at least it would be a good start until optimzation features come later.

Even starting with no customization hooks, and just global Package for Meteor APIs, and user code as plain ES modules, would be great. Any step that gets us onto on the native ESM path will be great.

It won’t be needed with native ESM, the native loader read exports fields.

awatson1978 · January 19, 2024, 12:21am

Wooo hoo! It runs! Excited to give this a test drive!

superfail · January 19, 2024, 12:24pm

ESM, please! It’s 2024!

denyhs · February 8, 2024, 1:35pm

Hello everyone!

Here is another update on what we’ve been working on in the past few weeks.

We fixed an issue where Symbol.asyncIterator on Cursor was not working. We updated the Reify package inside the core where it wasn’t updated yet. Also worked on issues like: Insert not working for collection created of the fly.

Right now, we are working on:

Uncaught Promise Rejection when throwing stub error
Check how difficult it would be to enable native ES modules on Meteor (We already have a plan for this. More update on it later )

We also moved forward with the docs, and it’s looking really good. Here is a sneak peek:

We are planning on releasing a Beta 1 pretty soon, so stay tuned!

You can follow the updates on our overview doc , and on this Github PR

denyhs · February 23, 2024, 4:04pm

Hello everyone!

Here is one more update.

We release one more Beta, this one being beta.4.

It includes:

The Async iterator now works (for await (const item of Collection.find()));
OAuth is now working as it should;
Meteor.userAsync now works in publications;
Isomorphism in Collection is back;

We’re already working on the tasks for the next one, so stay tuned!

You can follow the updates on our overview doc , and on this Github PR.

denyhs · March 11, 2024, 4:36pm

Hello everyone! If you are still not aware, a few days ago, we released our Beta 6!

You can check more about it here.

Your test and feedback is really important. See you soon!