New package "rocket:module" to provide CJS/AMD/ES6 Modules for Meteor packages with shared NPM dependencies

packages
1

Hey all, I’m working on a package called rocket:module.

I constantly find myself wanting to use libraries from outside of Atmosphere, and I think that having to wrap every outside library as an Atmosphere package is totally not ideal. I also always find myself wanting to use CJS/AMD/ES6 modules. rocket:module aims to satisfy these wants.

It’s not released or documented yet, but basically what rocket:module will do when finished is let you specify entrypoints using api.addFiles in a package’s package.js. In those entrypoints you’ll be able to use CJS/AMD/ES6 modules, and load things from npm for the client side or server side (not using Npm.depends, but a separate config for specifying npm dependencies that aren’t limited to absolute versions like with Npm.depends). Behind the scenes, rocket:module is currently using Webpack to compile bundles from the entrypoints. rocket:module will check across every entrypoint of every Meteor package in a Meteor app and code-split common libraries into a single bundle that will be loaded before all other bundles in the entire app. Any package on Atmosphere will be able to use rocket:module in order to specify entrypoints that depend on things from npm (and in the future perhaps other places too), and those dependencies of your package’s entrypoint bundles will be shared across all bundles of all Meteor packages in your app when those other bundles of the other Meteor packages depend on the same libraries. After that’s done, perhaps we can support entry points in the application level code too.

Any opinions or suggestions? Would it be useful for you? What plans or work are already in place in the Meteor community to solve this, if any? I’d like to know what direction is already being taken, if any, so that I can work with it instead of separately from it.

9 Likes
2

Brilliant! I’d absolutely use that. Anything that brings us closer to being able to use npm a) more closely like we can do it in just plain node and b) client-side (without sending anything to the client more than once!) is a fantastic step in the right direction.

1 Like
3

Can you describe some example use cases where we can benefit from using rocket:module vs. just using what Meteor currently provides (Meteor packages and Npm.depends)?

Can you also show an example of using the API of rocket:module. It’s ok if the API isn’t implemented yet :wink:

Does rocket:module solve on demand loading also? E.g. load package X only for route Y and Z.

1 Like
4

@Sanjo The first thing is that Npm.depends only works on the server side. If you’re only using Npm.depends on the server side, you’ll be limited to absolute package versions, and every package that uses Npm.depends will have it’s own version of libraries, so for example, if you have 5 packages that depend on lib@x.x.{1,2,3,4,5}, then that means you’ll have 5 instances of lib@x.x.x in your app (I made this github issue, but I think I remember seeing some more discussions somewhere, describing that each Meteor package gets all it’s NPM dependencies compiled into it instead of shared with other packages, here’s further proof). It’s a hassle to have to wrap non-Meteor libraries as Atmosphere packages just to use them; a waste of time and introduces high amounts of maintenance complexity.

I was thinking about on demand loading later on down the road map. The idea would be simple I think. When using api.addFiles, you can pass an option in the third parameter fileOptions (I made a PR to start documenting this parameter). Perhaps it can be called publicPath or something, and it’s value would be a path relative to your app’s public folder, then you can do whatever you want in your Meteor app code to load that file from the public location at any later point in time (for example, when flow-router executes a route).

Here’s what the basic usage for rocket:module will be like. In your package’s package.js, add an a file like normal, but it happens to be an entry point. The file name would end with “module.{js,coffee,ts}” (subject to change, but will be documented):

Package.onUse(function(api) {
    api.versionsFrom('1.1.0.2')

    api.use([
        'rocket:module'
    ], 'client')

    api.addFiles('client.module.js', 'client')
    api.addFiles('client2.module.coffee', 'client')
    api.addFiles('client3.module.ts', 'server')

    api.export('Lib', 'client')
})

client.module.js might look like this:

// Write your package code here!
var uppercase = require('upper-case') // from npm!

console.log(' --- ', uppercase('hello world!'))

// a Meteor export for other package to use if they depend on this package.
Lib = {
    hello: function() {
        return 'Hello.'
    }
}

If you want client2.module.coffee to be built into public (@glasser @awatson1978 can a build plugin build a file, place it in an app’s public so it’s public when the app is deployed for production? Perhaps we’d have to run meteor once in development before meteor deploy so that generated files meant to be in public exist):

api.addFiles('client2.module.coffee', 'client', {publicPath: 'path/to/public/place'})

Then in module.json you’d define your npm dependencies, similar to package.json:

{
  "dependencies": {
    "upper-case": "^1.1.2"
  }
}

Your entry points can require/import local files, and those local files can require/import other npm deps and local files, etc.

That’s basically it. rocket:module will handle the rest behind the scenes, sharing dependencies across all the entry points of all the Meteor packages in the application.

2 Likes
5

@Sanjo Another way to do the lazy loading of files instead of including them in the initial Meteor-made bundle might be to use @arunoda’s meteorhacks:picker for serverside routing, then get scripts as needed from there instead of from the public folder. As with the previous example, there might be a fileOption for that or something.

I debated between browserify, jspm, and webpack for compiling the entry points. I chose webpack because it’s the most flexible and configurable. Part of the usage of rocket:module will be that we can also put a webpack.config.js in a Meteor package, and rocket:module will see it and merge it into it’s default webpack config (on a per-package basis), so we can override the rocket:module defaults or add new settings. The defaults will be the css and babel loaders so we can import CSS files and use the latest ES features, including the ES6 Module and Class syntaxes. It will be documented that some defaults, like the output and entry options shouldn’t be messed with unless you know what’s happening in rocket:module, or else it’s likely to break. Perhaps in webpack.config.js, if the user returns a function instead of an object literal config, then we can pass that function compileStep as an argument so the user can customize the config more easily if needed, and then the function returns the final config that should be merged into the default (using lodash’s _.merge).

3 Likes
6

Yes, I have to agree… and something I have been looking for but haven’t found yet.

2 Likes
7

Hey guys, I have good news. I was really close to finishing rocket:module using the build plugin API that Meteor has had for a while, but the new Plugin.registerCompiler feature has landed in the devel branch! This is great as it will make what I was trying to do a whole lot easier and cleaner, so I’ll start refactoring to use this new plugin architecture (I won’t release a version of rocket:module written on top of the current architecture because I’ll be wasting valuable time on that effort, but I’ve learnt tons to apply to the refactor :smile: ).

3 Likes
8

Mmmmmm, Meteor’s new MultiFileCachingCompiler is going to be super useful for this!!!

9

I’m doing some experiments nowadays with React and what I’m simply doing is creating a .folder/ and inside it do all normal I do with webpack and tell it do build to client/app.js. It works great by the way.

1 Like
10

Ok, but then how do you use meteor packages ? include them all as globals ?

11

I think I don’t understand your questions.
Meteor packages are globals and are loaded before code inside client/. Take a look at this file:

12

Gabriel, if i understand you well, you build you app code with webpack in client folder. Then there are picked by the official build tool to be delivered to the client.
If you need a meteor package, you have to add it to the main app, where it becomes a global.
This solve only half of our problems since one of the point of modules/packages is to include dependencies (== external packages/modules) ONLY for the part of the code (specific module/package) that need it.

Let’s say that i have module A, module B and module C in my app, coded as ES6 modules.
Now, i find that module B would need FlowRouter. I should have a mean to include/require the FlowRouter as a dependency of module B, but without introducing it as a global for the whole app. With your technique, i think it would not be possible.

This is were it becomes tricky, now that meteor has official “packages” to make them work with ES6 modules flawlessly. I wonder which solution MDG will come up with, perhaps it could be better to discuss it without it’s too late (= already carved in stone).

13

Oh I understand now. You want to make a “closure” when you require packages so those packages do not leak the global scope, right?
Something like:

import FlowRouter from 'meteorhacks:flow-router';

// FlowRouter is available only in this file's scope.

Maybe you could try removing all packages from global scope and implementing that import thing on top of window.Packages? (A naive solution)

14

I’ve just done the Meteor package globals to SystemJS modules mapping. I guess something similar can be done with Webpack. https://github.com/Sanjo/meteor-es2015-modules-and-di/compare/340cf3a66ac5fdccc7dc1215eaa71b155ba0d3c6...38ca1e7674353696762ee7e72856e65812631c2b

1 Like
15

That looks awesome does it work?

16

Yes, it works so far. :smile:

1 Like
17

That’s where my idea for rocket:module originated from, I was doing the same. With rocket:module you’ll be able to do the same, but meanwhile also sharing dependencies and code across packages and the whole application.

18

As far as I know, MDG doesn’t have an official module solution yet. I’ve only seen the syntax support with the ecmascript package. The new Plugin.registerCompiler API (released in Meteor 1.2) is amazing, which is what I’m using for rocket:module to handle dependencies and shared modules across packages.

19

Yes, but since it is on their todo list to get ES6 modules support, i wonder how they will solve this.

1 Like
20

Yes indeed it can. I’ll look into this with rocket:module after the initial set of features is complete.