Experimental Code Splitting 🔪📦

I made a little (and hacky) package that implements basic code splitting for @benjamn’s proposed syntax with async functions and nested imports (see this issue and this explanation).

GitHub repo:

How it works:

If you want a module to be loaded on demand and not when the page is loaded, you import it inside an async function:

async function loadAsync() {
  import stuff from './module.js';
  // Do something with `stuff`…
}

loadAsync();

The compiler in the package recognizes that ./module.js can be loaded asynchronously and puts it in a separate JavaScript bundle. This additional bundle is only downloaded when loadAsync is called.

Examples:

There are a few examples in the repo, including one that shows how to use code splitting with React Router. If you want to try them out, just clone the repo and run meteor npm install and meteor in the example directories. In development mode, the package logs messages in the browser console when bundles are requested and loaded. Additionally, you can see the loaded bundles in the sources and network tabs.

Note that this is just experimental! You shouldn’t use the package in a real application because it’s missing some important features that cannot be implemented with Meteor’s build plugin API (see the GitHub repo for details). Also, there’s a good chance that I missed some details and special cases in which the package isn’t working correctly. :smiley:

31 Likes

Cool stuff! Minor heads up - the syntax was rejected in favor of a function-like operator:

const variable = 'String';
const module = await import(`Template${variable}.js`)
const module2 = await import('RegularString.js')

It is currently stage 3


4 Likes

Thanks for that link! I didn’t know that this is already at stage 3. I still like the declarative syntax more, to be honest. :sweat_smile:

1 Like

Really excited that Meteor is catching up with state of the art in module bundling!

2 Likes

Well, his approach is not standard though. With import() being in stage 3, I guess we’ll be able to use this via babel soon. Do you plan to support import() once it’s in babel?

I mean it’s just a first pass - I’m sure the next version will follow the standard and be better in lots of ways! Also, import() doesn’t have anything to do with Babel, since it’s not a syntax feature but a bundler/runtime feature.

2 Likes

I like the declarative syntax as well, but will take any API to support this! :heart_eyes:

1 Like

@klaussner This is really cool! Thanks for taking my idea and running with it, even though I think it’s true that code splitting in Meteor has to support import(...) as well as (or perhaps instead of) import declarations in async functions.

One thing that still concerns me about import declarations in async functions is that they introduce an implicit await at the beginning of the async function, even though async functions are supposed to execute synchronously up to the first await (kind of like how the new Promise((resolve, reject) => ...) constructor executes the resolve/reject function immediately).

Inspired by your work, I spent some time in our “hack week” this past week building an async bundling system to support import(...), which makes me optimistic about these features landing in Meteor 1.4.3.

One lesson from that work: import(<dynamic expression>) is less appealing than it might seem, because it’s a security risk for clients to be able to import modules you haven’t authorized in some way. One way to authorize imports is to list your entry points explicitly (e.g. in webpack.config.js). Another (much better!) way is to require that import(...) always takes a string literal, so the set of authorized modules is statically implied. In other words, import(...) ends up behaving a lot like require(...), in the sense that require works with dynamic expressions if the same module is required with a string literal elsewhere, but arbitrary dynamic module identifier strings won’t work.

One more question on my mind: does import(...) support make any sense on the server?

16 Likes

Thanks @klaussner and @benjamn for this

My initial hunch is no. A server has to have everything ready for all users. As node is a single threaded process, it could reduce experience for other users (not just the current one). Besides, Meteor does not take that much memory so we can say that it’s free server-side. Also, every time you update the server, you risk having to re-upload all the libs again.

3 Likes

Wow, that sounds great! :blush:
Regarding import(...) on the server: I think it would be useful to support this for code that is shared between client and server, e.g., for Meteor methods:

Meteor.methods({
  async rarelyUsed() {
    const bigModule = await import('./big-module.js');
    // …
  }
});

I would expect this to work everywhere. On the client, big-module.js would be loaded from a separate bundle and on the server, await import(...) would be similar to require.

2 Likes

this is awesome! thanks for the PoC

1 Like

Absolutely amazing work @klaussner. With all the work from @benjamn’s, the TC39 discussions and so on, it looks like you guys are nearer to solve the puzzle than ever before…

In case you didn’t hear about, somebody from Airbnb started something in November:
https://github.com/airbnb/babel-plugin-dynamic-import-node (at least server-side, pretty rudimentary stuff though)

But maybe I can give another perspective to code splitting:

I think we can all agree on that loading big modules asynchronously (in a browser env) is one of the biggest weaknesses of Meteor’s build system compared to Webpack at the moment, right?
But for me, far more important than loading ./big-module.js would be loading complete packages like ‘meteor/big-local-atmosphere-package’ or ‘big-npm-package’.

Doesn’t Package.js already provide us with all the needed information for code splitting like which module to rely strong/weak on and which deps just to use? If you look at rocket.chat or other big meteor projects, what is currently developed with the “umbrella” package approach is basically a folder-based recipe on how to split the app into logical modules perfect for code splitting, or am I wrong?

Personally I think loading something from packages/ in an async way would be extremely valuable than just loading a file in the project dir. I know, MDG wants to move away from Atmosphere in the long term, but when it comes to splitting a webapp into big modules I think the whole npm link / eslint / babel compile horror is just not worth the time so people start to structure their app inside the imports/ directory just because they fear that local atmosphere packages get deprecated in the future :slight_smile:

Oh, and that’s what I also found googling around / some literature:



@benjamn, are we waiting for tc39 proposal to firm up before we implement? I am super excited about this. Not only for our apps, but for Meteor as a whole. The build system is a key advantage.

Also, since we are in the process of adding it as a new feature, I would love if we could properly support service workers. What I mean: if a bundle has not changed, why change its name / hash and force a reload? Keep as-is so the client loads it from the cache. Only change names of bundles that indeed have a change.

This would be a game-changer for Meteor and the ecosystem. As it makes apps loading very smart, only loading what physically has changed.

3 Likes

I’m working on support for dynamic import(...) as we speak!

If all goes according to plan, the next 1.4.3 beta will include a working prototype. That’s part of why there hasn’t been a 1.4.3 beta in a little while.

Because import(...) is asynchronous, it definitely seems amenable to service worker-based caching. However, there’s no necessary reason fetching modules has to happen via HTTP (I’m currently planning to do it over a web socket), so the service workers API might not be the only option for async module caching. There are multiple places we could intercept module fetching for caching purposes, even if we didn’t have service workers.

Looking forward to showing you what I’ve built!

41 Likes

hey benjamin, that’s SO COOL! I’m just wondering if there’s a way to define how things get built with Meteor? I had to get off Meteor recently and go to Webpack unfortunately because creating a Chrome extension with Meteor was difficult. Code splitting in Meteor really excites me, would love to get back to Meteor in the future with this progress :slight_smile:

1 Like

I think it can be hard to find the client-side build output of a Meteor app since it’s hidden away inside various directories, did you try loading that JS file into an extension?

I think using Meteor to build client-only JS bundles should be possible but probably harder than it could be since it’s not the main focus.

Ah, so again, should it be in a “minor” version?:wink:

1 Like

You are my hero, and a great inspiration to follow !

1 Like

You made our day. Thank you!

1 Like

Amazing!!! … this darn forum only let’s us click on ‘like’ once. You would have been overwhelmed with ‘love’ :slight_smile:

2 Likes