Modernizing Meteor's Bundler

announce
1

This post aims to initiate a discussion on enhancing the Meteor bundler by aligning it with industry standards, adopting external tools, relying on open-source maintenance, and revive Meteor within the JavaScript ecosystem.

Motivation

Meteor remains a robust framework for:

  • Facilitate your app ideas with a versatile toolset unifying technology using the web for consistent multiplatform experiences
  • Promote code reusability through opinionated and isomorphic data management
  • Scale real-time app development with easily implemented reactivity
  • Utilize third-party packages to expedite app feature delivery

Despite these strengths, Meteor lacks modern bundler tools to:

  • Ensure smaller app bundle sizes and enable tree shaking
  • Ensure compatibility with modern modules (ESM)
  • Flexible build configurations and common tools support

These implementations have long been desired by the Meteor community. However, previous attempts to integrate them into the Meteor core faced challenges, leading to project abandonment (e.g tree shaking). The Meteor bundler lacks of common understanding and documentation within the community, diverging from adopted standards, making any bundler feature implementation to be really hard to accomplish by core team or community members.

Adopting a modern bundler and standard build mechanisms offers several advantages:

  • Ensures app build compatibility with standard build procedures, external tools, plugins, and extensions
  • Encourages open-source expansion, as adopting to standards allows contributions from the broader development community
  • Attracts more developers and companies preferring standard and robust approaches for app development

Exemplified approach

Experimentation has led to a simplified solution for adopting a new bundler that aligns with modern standards. The approach is illustrated in two Meteor example apps: simpletasks-webpack and simpletasks-vite.

I already shared with the community this approach on a previous forum post, Meteor + Webpack integration - Tree-shaking, Secured code and more.

The approach involves:

  • Using a modern bundler to watch and compile the app code to generate a intermediate bundle of the app
  • Compile the intermediate bundle with Meteor and inject Meteor’s core and atmosphere packages into the app code

image
image508Ă—595 7.77 KB

This solution combines the strengths of both worlds:

  • With a modern bundler:
    • Support for tree shaking
    • Default support for ESM
    • Compliance with standards for code splitting management (aka dynamic imports)
    • Fast incremental buildings with bundler’s cache mechanism
    • Compatibility with various extensions and plugins for bundle control and monitoring
    • Enhanced and maintained building implementations
  • With Meteor:
    • Full benefits of the platform, including data management, isomorphism, reactivity, utility packages and native integration.

Some Meteor-specific bundler capabilities, like code splitting and lazy imports, are sacrificed, at least for app direct code. It is supported it indirectly using meteor packages.

This approach simplifies modern build capabilities without struggling with the Meteor build system, plugins, or other historically problematic approaches, providing a stable solution over time.

What would mean of Meteor core adopting a modern bundler?

Meteor’s core adopting this approach in practice means that the internals remain untouched. The existing core still handles resolving its core and packages, while a new modern bundler is introduced in the Meteor core as an independent layer to resolve the app code.

This implementation aims for ease experience for developers. The “meteor create” command would ideally remain unchanged, providing an easy setup experience without apparent differences from the new implementation. The “meteor run” command would simultaneously utilize both modern and meteor bundler, providing clear development feedback of the processes.

Additionally, a new meteor.config.js file would extend configuration for the included modern bundler (whether Vite or Webpack) as well as any Meteor bundler specifity we may want to introduce, allowing easy and extra customization of bundler for any Meteor developers.

Certainly, challenges may arise when integrating this with Meteor toolset, but it appears less complicated than directly modifying Meteor’s bundler internals.

Feedback

What are your thoughts on incorporating modern build features to Meteor by adding a new bundler, and making it as a core solution?

10 Likes
2

A practical case: billdroid.app

I’ve implemented this approach in my recent indie app, billdroid.app. It is already in beta, in a production-like environment.

This app served me as an opportunity to use a stack expanding the limits of Meteor, and a practical implementation of a receipt scanner for financial management using more modern multiplatform procedures with CapacitorJS. It utilizes Webpack as a robust bundler and adopts Meteor extensively for rich features. I plan to showcase the benefits attained in a separate forum post, emphasizing that embracing a modern ecosystem requires conviction, simplification, delegation, and determination.

Key benefits in this app include tree shaking, ESM, cache for fast builds, image optimization in bundle compilation, i18n management using lingui webpack’s plugin, easy service worker and PWA setup with Workbox webpack’s plugin, and more. CapacitorJS is incorporated similarly, serving as a CordovaJS replacement.

2 Likes
3

In my opinion the perfect setup would be integrating with Vite. Go all-in on modern browsers features and serve modules as ESM individually

7 Likes
4

Conceptually it seems like a smart idea.

Can you expand on what would be sacrificed here with maybe a couple examples? What does supported indirectly mean?

Would there be any other potential downsides? I mostly see only benefits so just curious what I’m missing.

5

This would be a decision to make.

On my tests using “webpack vs vite” with this approach, I have found for instance that Webpack’s cache system is more advanced and produces incremental rebuilds faster, like on my machine ~64ms per change. In contrast, Vite gets like 1-2 seconds for the same action. This could be the current state of Vite on the vite build command that by default disables optimizeDeps. I enabled it but resulted on a harder configuration and without any significant gain on performance. Maybe the idea of being an experimental feature on build command is the problem. I understand Vite dev tool works better, however this approach relies on vite build since Meteor is still the dev tool in charge of setting up the local environment.

Webpack 5 has worked really well for me, I think is more mature and robust, even though is no longer so hyped or popular. Next.js is still the bundler that uses behind the scenes.

I would like to experiment more, but I think is important to keep a good dev experience on the tool and approach we would adopt.

2 Likes
6

I explain more about the downsides on the repository itself, here.

Anyway, what is lost is basically two things:

  • The way Meteor has to resolve dynamic import() using DDP protocol behind the scenes in favor for the standard one using Webpack/Vite, which is the same idea but serving the splitted modules via HTTP.

  • Lazy/Nested imports. The es6 import statements within inner blocks of your code (a conditional statement, a function body, etc), like if (xxx) { import xx from 'XXX'; }. This is not a standard on bundlers and actually is what breaks any portability of Meteor projects outside or any external tool adoption for your app code, like Jest, Storybook, or any other tool that enhances your development experience someway and needs to compile your code.

However, there is still one way that you can use these Meteor’s features, it is by adding this logic behind a meteor package. Since Meteor bundler is still the one acting on meteor/* packages, it will recognize and apply its logic effectively. This is what I mean for indirectly supported.

1 Like
7

Can you expand on this? What does it mean as being sacrificed but supported indirectly using meteor packages? An example might give more clarity about this.

Exciting updates. Thanks

Edit: just realized that my questions were answered on the succeeding posts

1 Like
8

Wouldn’t that mean that we need to still maintain Meteor bundler and build tool for packages though?

2 Likes
9

Is there much difference between DDP and HTTP for this particular use-case?

1 Like
10

Very nice plans. One thing to keep in mind here is that, as far as I can tell, choosing Vite would also make the Vue-Meteor integration a lot more straightforward and easier to maintain. For example, in the current Vue 2/Meteor integration about 25% (rough ballpark, citing from memory here) lines of code implement the tracker and around 75% deal with making things work with the bundler. For Vue 3, all of the latter could be left aside it seems as it already works well with Vite. The same could probably be said for other view libraries as well.

Take my comment with a grain of salt, though. I have not looked into this topic very deeply. So there might be surprises. But it should not be very much work to go over the most popular view libraries and see how well do they work with Vite out of the box.

2 Likes
11

I’m not opposed to updating to a modern bundler, if we can add everything back in (isomorphism, reactivity, data management, etc).

We don’t really use code splitting or lazy imports all that much; and to be honest, I have a webkit template that I’ll sometimes build UI interface in because it’s so much lighter.

But there is one feature/issue that would be great to get fixed:

When compiling React components in Atmosphere packages, the bundler will create some sort of root Javascript object that it attaches all the subsequent components to; and when it rebuilds the code in the final codebase, it attaches them via a React component instead of a React function; which breaks the Rule of Hooks and all downstream pure functions.

What winds up happening, is that you can’t pass a prop or children elements from a root (parent) component defined in the main app into a pure functional component defined in an Atmosphere package. There’s a sort of cleaving, where all the React functional packages defined in Atmosphere packages get attached to a React component, breaking props passing through the functions from App code into Atmosphere code.

That has been my single biggest headache with the bundler for the past 4 or 5 years.

That being said, if we adopt a new bundler, I’m sure it will introduce new headaches.

2 Likes
12

Indeed, Meteor bundler remains responsible for compiling meteor/* packages and integrating them into the compiled bundle through the modern bundler process.

Additionally, Meteor would focus on minimizing core and package sizes to reduce the basic bundle weight. I have seen already a lot of work around this reduction or to remove stale libraries used on core. Anyway, a migration to npm where possible would enable tree shaking reductions at the app level.

2 Likes
13

DDP

The DDP approach in Meteor creates dynamic modules per submodule path from the module used in the import() statement. These modules are served on-the-fly as strings in response to client requests, providing advantages like live updates and efficient handling of increased dynamic imports.

The downside is that it relies on the DDP responses, requiring a server connection for proper serving. This makes offline support for dynamic bundles impossible and lacks built-in support to include these dynamic bundles in the native compilation.

The next picture illustrates a response in the simpletasks app of a Meteor dynamic import when loading the tasks page.

image
image1680Ă—562 37.2 KB

HTTP

HTTP is the main communication protocol for the web. Webpack and Vite compile dynamic modules into single files or can adjust granularity via configuration, matching similarly as Meteor’s behavior likely. Direct file generation enables inclusion in native compilation, supporting offline availability, using service worker caching for faster module loading, CDN, and all tools and mechanisms implemented on the top of HTTP.

Opting for a standard approach used widely in various companies and projects globally minimize any disadvantages. The reactive aspect in Meteor may be the only drawback I can think of, but it doesn’t seem to me a life-changing implementationat all.

The next picture illustrates a response in the simpletasks-webpack app of a standard dynamic import when loading the tasks page.

image
image999Ă—289 24.7 KB

2 Likes
14

It’s important to support any view library embraced by Meteor developers. Although I lack experience with Vue, I’m up to examine a basic app and explore the approach in both Webpack and Vite, gaining a better understanding of your concerns. A quick search showed me an existing guide on supporting Vue 3 in webpack with loaders.

Any recommended Meteor-Vue projects for evaluating modern bundler support?

Anyway, I envision the modern adoption process into Meteor as flexible enough to switch from one to another through a configuration field in the meteor.config.js file, allowing users to choose their preferred bundler even. However, Meteor should prioritize one and consider expanding or replacing it when a switch becomes beneficial or to cover more scenarios as the one you mention with Vue if not possible with the other.

1 Like
15

I’m not opposed to updating to a modern bundler, if we can add everything back in (isomorphism, reactivity, data management, etc).

That is surely perserved by delegating the compilation of Meteor specific characteristics to the Meteor bundler itself.

We don’t really use code splitting or lazy imports all that much; and to be honest, I have a webkit template that I’ll sometimes build UI interface in because it’s so much lighter.

There are large projects urgently require bundler reduction. Standard implementations with modern bundlers are a clear win here.

When compiling React components in Atmosphere packages, the bundler will create some sort of root Javascript object that it attaches all the subsequent components to

Atmosphere packages may encounter issues, making users to use direct file references rather than relying on packages. I have experienced this on the past actually.

That being said, if we adopt a new bundler, I’m sure it will introduce new headaches.

I know. I would say here that any thing related with our profession brings up headaches :rofl: We are problem-solvers by nature.

Switching to a modern bundler expand our knowledge base by going into a larger pool of company and developer experiences. Meteor’s current issues often require a few experienced developers to navigate through outdated, undocumented procedures, as well as reimplementing things that are largerly managed outside our smaller community.

I wouldn’t be afraid to embrace the new when the community claims for modern procedures, rather than persisting in a struggle with the old.

4 Likes
16

Good take.

Regarding options, does the developer have the option to opt out of using a modern bundler and stick with just the Meteor bundler?

2 Likes
17

Definitely, the meteor.config.js should extend configuration for both modern and meteor config. Consider adding a bundler field to choose between them to build the app code, favoring the modern by default.

2 Likes
18

Until now, dynamic imports cannot be piped through a CDN, making it dependent on main app server bandwidth (slow and expensive).

3 Likes
19

Both Vue and Svelte have standardized on Vite. For Vue 3, their tooling guide recommends using Vite over webpack, which was used for Vue 2 previously. And Svelte’s docs recommend SvelteKit, which is also based on Vite. So if Meteor would choose webpack, then the tradeoffs for this should really be determined before making the decision.

For Vue and perhaps other view libraries as well, I think it would be best to have a quick chat with @akryum who has maintained the Vue 2 integration with Meteor and is a member of the core Vue team. And maybe also with the author of this repo. I would assume they’d be able to lay out the pros and cons of both choices pretty well as far as view libraries are concerned.

But in any case moving to a widely used bundler would be a very good step towards aligning Meteor with the general JS ecosystem again. Instead of reinventing the wheel (view libraries, bundlers) it would be much better for Meteor to just use the industry standard and focus its own efforts on just what makes it special (pub/sub, methods, accounts system etc).

I looked it up. To currently use Vue 2 with Meteor, you need the the following two packages:

The first one makes the tracker work in Vue components (so gets you the special stuff of Meteor, like making subscriptions etc) and the second one integrates files with the bundler. The first is only around 250 lines of code, while the latter is around 10x more. Ideally Meteor whould be in a place where the bundler issue is already taken care of, since an industry standard bundler is used, and so integrating a new view library with Meteor would only be a 250 lines of code type of affair. Easy to create and easy to maintain. One can hope :slight_smile:

5 Likes
20

Meteor switching from Isobuild to Vite would be a clear win for compatibility with the rest of the ecosystem and would make Frontend frameworks integration very easy.

8 Likes