Ideas from MDG: Meteor 1.3, the Meteor Guide, and ES2015 modules

core
1

TL;DR we want to write the Meteor Guide using ES2015 modules, we need your help to find out what needs to be done in Meteor 1.3 to make this possible. Look at the list below and see if there are some concerns we missed!

Background

If you’ve been following along with the Meteor Guide, you know we’ve basically been writing it targeting Meteor as you can use it today - this means Meteor 1.2.1 and all of its flaws. In this spirit, we have been writing the app structure guide and example app in the packages-only style: all of the app code is organized in Meteor packages, which are currently the only way of achieving “modules” in a Meteor 1.2 app.

However, Ben’s working on adding first-class support for ES2015 modules - require, import, export, and friends - which will be shipping as the big new feature in Meteor 1.3.

Given this and the challenges around building an app out of Meteor packages, I think @tmeasday and I have been considering writing the initial version of the guide targeting ES2015 modules instead of Meteor’s classic package-scope variables, load order, and eagerly-evaluated code.

To that end, today we had a productive discussion with @benjamn, @debergalis, and @zol to see what it would take to be able to ship Meteor 1.3 with modules and the first version of the Meteor Guide together.

What would it mean to have the first Meteor Guide target modules?

In order for the first version of the Meteor Guide to target ES2015 modules instead of the classic load order approach or package-first development, we would need to be absolutely sure that this is the best way to build apps in Meteor as of 1.3. The only way that can be true is if there are no hidden blockers for doing so - we need to make sure this stuff works with all of the crucial packages, patterns, and frameworks people are using to build their apps.

Topics that need to be discussed and resolved to write the Meteor Guide against ES2015 modules

In our meeting we basically went through the points one-by-one. I tried to write down the main points for discussion.

  • Testing: Realistically we need a sensible way to unit test a module in a reasonably isolated way.
    • Ben: Perhaps testing just comes down to not eagerly-loading application code, and just loading the modules you want to test?
    • Tom: Whole app testing might not actually be related to modules. We should just find out if we can simulate the meteor test-packages experience with a module. This is basically “compile the package as usual, but also include the onTest files and dependencies”.
    • Ben: We could just have a “sub-app” in test/ that is loaded when you are running tests, and those could have modules that are test-only.
    • Could we have .meteor/packages just specify which packages are test-only, dev-only, etc. For example, you could have mocha as an app dependency, but only in test mode.
    • You’d also have an eagerly-loaded test runner where you can register tests to be run.
    • meteor test could accept an argument that filters the tests based on the name they were registered with, or the module in which the tests are defined. Should also start a test database, etc.
    • Matt would like to see the hypothetical guide article that outlines how you write and run your tests in the module world. Then we can make sure the module system supports all of that.
    • Tom will write the testing article against an idealized module system, if we ship the guide before modules ship, we’ll just omit that section or put it in some “future” area.
  • Blaze: How do blaze templates, which don’t have any concept of import/export, work with this? Can you co-locate templates and their JS logic?
    • We could make these lazily evaluated, and only be executed when you import them
    • {{> templateName}} means Template.templateName. In module world, do we need to first import templateName from ‘../module/templateName.html’ somehow? Answer: no, given the other discussions about Blaze/React, it’s not worth making major changes to the Spacebars syntax at the moment.
    • Sashko/Matt say - it’s OK if all template files are eagerly evaluated and work as they do now. Sashko says it’s important to him that all code can be put in the imports directory, including templates, so that we can make the code structure nice for the guide.
    • For further discussion: How important is it that feature code (JS, HTML, CSS, images, etc) can be co-located?
  • Build plugins: Do build plugins work as-is? how do you use modules with CoffeeScript, does this open up TypeScript? Have we compared what we are building to what is needed by the Angular/Angular2 community? What about CSS pre-processors?
    • The dependency tree is analyzed based on the output code - so as long as the CoffeeScript compiler outputs require(‘myFile.coffee’), we’ll detect that dependency.
    • LESS does all of its own importing and exports CSS, which could either be added as a style tag or a JavaScript resource. The CSS pre-processor stuff isn’t affected by ES2015 module work for Meteor 1.3, and will work just as before.
  • Meteor package system: What’s it for? Should people start shipping reusable Meteor code on NPM as of 1.3?
    • Meteor packages are really good at doing totally different things on the client and the server - you can set up the full stack just by adding the package
    • For example, for DDP, you’d need to import ‘ddp/client’ on the client and import ‘ddp/server’ on the server if you want different functionality, but in a Meteor package you can just use ddp and it does the right thing.
    • client/ and server/ directories in NPM packages don’t have a special meaning at the moment, but they could - or we could have a special package.json format. Probably won’t tackle this for 1.3.
    • If you don’t have the above special build system requirements, there is basically no reason you need to publish your package on Atmosphere over NPM - this also removes the need for robotic wrapper packages that just re-bundle the same code that’s already on NPM.
  • Assets: What about assets like fonts and images? In the current package system, you can organize them in package.js, but in a Meteor app today you’d have to put everything in public/. What do you do in module world?
    • Modules are about loading things that compile to JavaScript code
    • Images aren’t something you would ‘require’ since there is no JavaScript content involved
    • Perhaps you should be able to stick images anywhere in the app (not just public/), and have a sane way of referring to them
    • Idea from Sashko: Perhaps we want getPathToAsset(‘../relative/path’);, so that you can put images next to templates and refer to them without having to know the absolute path to the containing directory.
  • Apps with multiple entry points/UIs/services: Right now, you can have lots of packages and lots of apps, is this possible with modules? Do you need to make modules into NPM packages for this? Are Meteor packages still the way to go? If so, do you still need to list all of the files?
    • Didn’t have time to address this in the meeting.
  • Cordova: Can you import Cordova plugins?
    • We don’t yet have a good mental model here yet. It would be nice if you could import a cordova plugin, and make sure the necessary side effects run.
    • These things can work like “legacy” meteor packages - the dependency scanner doesn’t need to understand them. This is one advantage over using Webpack in Meteor 1.3 - it assumes everything is a node-style JavaScript module.
  • File structure: Do we want to keep the imports directory convention proposed in the PR?
    • Ben says backwards compatibility with the existing loading logic was important to him, and imports is just one way to get that.
    • Other ways are special file names, special comments, etc.
    • Matt says this directory should sound more “default” and less “opting into a fancy new feature”
    • Ben said one big reason was that the LESS package picked this name. modules is another possibility, feels like node_modules. This is kind of a bike-shedding topic overall.
    • We could also have an app “control file” that just specifies entry point. Either a JSON file we can statically read, or a JavaScript file that just imports stuff inside conditionals somehow.

Please give us feedback!

What concerns do you foresee in the world where a Meteor app is built our of ES2015 modules? Did we miss anything in our discussion/analysis? Please let us know!

Thanks,
Sashko

17 Likes
3

The answer to 'Can you import Cordova plugins?’ is basically no. Cordova plugins use their own configuration, bundling and installation mechanisms. Although the Cordova tools actually bundle your plugin code into some kind of AMD-like module structure (and it has its own asynchronous loading mechanism using XHR), it then takes the resulting module and injects it into a scope specified in the plugin.xml (that usually means just putting it on window, but it can be more complex). Plugin installation also includes copying over native code and modifying native project files according to rules specified in the plugin.xml. All in all, without changing Cordova itself I don’t think a model based on imports makes sense here.

1 Like
4

Very happy to see this being considered and discussed.

In the short time I’ve been developing with Meteor, I’ve seen a fundamental shift away from being a (mostly) isolated ecosystem towards embracing the best of that which the rest of the JS world has to offer. This is another important step in the right direction. The more open Meteor becomes, the more relevant it will be.

Opinionated, not obstinate, as it were.

5 Likes
5

How about creating an autoinclude package that includes everything in the package or app. It could be added to apps when updating to 1.3 and will make it easier to start using meteor without the need to understand the inclusions and requirements etc… You know just like the autopublish package.

2 Likes
6

That strategy doesn’t address the migration path though - you need to be able to have some parts of your app loaded in the old style and some parts loaded as modules so that people with apps can migrate.

1 Like
7

The best strategy i’ve found is to use ES6 with de-structuring to mimic how modules work
It also forces you to use a module pattern instead of potentially using more globals, which leads to better code in the mean time.

For example:

// utils/posts.js
const PostsUtils = {
  belongsToUser(doc, userId) {
    return ...
  }
};

this.PostUtils = PostUtils;
// end file

// some other file
const {belongsToUser} = PostUtils;
// use the function
belongsToUser(doc, this.userId);

Then when it comes time to migrate, hopefully all your code looks like the above. Then swap out the top and bottom of the file

// utils/posts.js
const PostsUtils = {
  belongsToUser(doc, userId) {
    return ...
  }
};

export default PostUtils;
// end file

// some other file
import {belongsToUser} from PostUtils;

belongsToUser(doc, this.userId);

or if you prefer individual exports you can tweak your module code to have an un-named module and keep the same API for the consumer:

// utils/posts.js
export belongsToUser(doc, userId) {
    return ...
}
// end file

I was able to do this with a few Vim macros and couple of beers for when I was upgrading to webpack and the overall strategy worked out well for me.

5 Likes
8

Alright, I didn’t think of that. How about just for new apps created with 1.3?

9

I like this. @sashko - if we end up not including modules in the first version of the guide do you think we should recommend an approach like this?

1 Like
10

I think we should be able to designate a folder that loads the old way or something like this. There has to be a way to transition smoothly. Both the old and new way must be supported at the same time somehow. Every which combination too:

  • previous app with automatic load order, and you want to start controlling the load order of some files
  • new app where you are controlling the load order, but then want to designate an area (or multiple) that use the old automatic load order
  • an app with load order completely automated the old way
  • an app where you control the load order for every file

We should do anything we can to find any way possible not to have 2 concepts of packages, and of course settle on NPM packages (phased out over time of course). I mean, would the original Meteor packages become the new concept for modules?

do the same you guys are doing with Blaze–throw it out.

I think finding a way make Atmosphere obsolete, and let everything live on NPM would be amazing for marketing and bringing Meteor back into the greater Node community at large. Obviously this would need to be phased.

…Perhaps we can specify how to make isomorphic packages through webpack. I mean that’s the obvious idea. Basically this plus the need for Fibers has been the main things killing Meteor’s marketability to the rest of the node community that doesn’t wanna feel so locked in. Obviously MySQL and dbs being another, but that’s a different ballgame–that’s more a matter of just adding more features, rather than changing the core of how Meteor works. As an aside, I also wonder if when await and async come if they could be used instead of fibers. The server side async vs. fibers play is another marketing angle that needs a new sheen. I feel like somehow we could offer async APIs that look almost like express, almost as a facade, just to not feel so foreign to those Node developers that stuck their nose up at Meteor. We’ve never revisited the Fibers vs Aync debate since the early days–but does the API opened to developers in Meteor.methods and publishers even need to be syncronous anymore? And for the app starting up, couldn’t we make it possible to start it up in a way that replicates how you do it with express, req, res, etc, etc. I’m just saying could we not build an async API on top of all the internal sync tools that make it feel like starting any other Node APP to non-meteor developers.

I think really that’s the heart of the issue we have here: the cats out the bag–everyone knows Meteor doesn’t play nice with the rest of the node and javascript world. It’s minor league the Flash of the Node world. See what I’m saying. We should do anything and everything to expel that notion, even if it’s just marketing, and extra features that the rest of us don’t need. We should feel just like anything else you’re doing in NPM, as much as possible, and there are now low hanging fruits for us, whereas before this would have been way more difficult and not necessarily right for business. For example, everyone knows how to use module imports and webpack these days, and now you’re going to undo a decision that was made long ago not to use NPM. I’m not saying that wasn’t the right decision then–it was. Everything panned out with perfect foresight on MDG’s part. MDG basically helped a bunch of developers to get this point where the rest of the tooling would catch up. Now we gotta rejoin the main branch on this matter. And we should on as many as possible, and recognize that as our goal.

We need to look like a standard Node app. I’d say “again,” but we never did. But now it’s crucial. We at least need to pretend. Until we do we’re gonna continue scaring people away. Similarly, we need to continue to unbundle so this hypothetical Node developer can dive into Meteor without worrying that he’s making such a big commitment. This is where we failed with Blaze, and one reason (of many) Blaze didn’t catch on like React did. Next: that brings us to a major “in” for us: React. But not where you might think. People using React, who are unsure of their backend, can now choose Meteor. SO, the main/prime/initial use case we wanna target here is: selling that person. Before, Meteor wasn’t even an option. React wouldn’t work with it–so they couldn’t use Meteor. Now, anyone who starts top down (i.e. with React rather than Node as their main commitment) can pick Meteor. WE WANT TO BE AS EASY AS POSSIBLE FOR THEM TO USE METEOR AS POSSIBLE. That should be your main business objective–not just adding modules support for the sake of adding module support. They should be able to add all of Meteor like any other NPM package basically. Hell, they can setup an express server that uses our own duck-typed mocks so they feel like they’re doing the same they always do. Then they can go on their marry way using React (perhaps our meteor-enhanced version of it). And then at just the right time we introduce them to server side publishers and Meteor.methods–right when they need them–and in a way that isn’t so foreign, but perhaps is slightly adjusted to feel more Node like. Maybe it doesnt need to be. But I feel like starting up the server and adding Meteor needs to be something you can do like any other minor express-based web app.

What i’m saying is there is a sales funnel here. The first things you do need to be just like anything else you do on/with NPM/Node. Only as you go farther down that funnel do things get more foreign to you, i.e. the meteor way proper. You’ve already conquered one with the decision to go with React and the simple mixin you already built to make it feel more react-aligned. And that’s either the biggest one or the second biggest one. The next is modules. And I think its big picture goal is to take you down the path to looking like I just described. Meteor deserves to be a lot more popular than it is. Look at the shear amount of javascript information coming out on the daily vs the amount that is specific to Meteor. So it’s more than clear Meteor isn’t getting the credit it deserves. Sure, it’s achilles heal is that it forces you to lock into various server side technologies, but enough time has gone by that I just get the sense that something is wrong. Many developers several times over now have written Meteor off, finished the project that could have used Meteor, and started yet another project without Meteor–when on that second project, Meteor might have been a perfect choice. Meteor got a lot of work to do in the marketing department. It’s not throwing its weight around enough. For our various shortcomings, we’re going back in our hole, and saying: “hey, that’s not my specialty.” We’re doing that too much.

I guess now’s the season where all this changes. And it does seem like you guys are prepared to make the hard decisions. We likely have many more. We should make em, and provide conversion/transpilation tools as much as possible to aid in the process. That’s opposed to doing the 80% good-enough solution that appeases both parties. We should pick the forward-seeking solution whole heartedly, and offer as much help as possible to bring developers’ old code up to speed, whatever that help might be.

5 Likes
11

Totally on board with this. Thanks for the detailed feedback!

1 Like
12

I don’t think you’ll find any argument that moving to modules and npm packages is the right way to go. In terms of migration path for existing packages, I’d like to see proper Meteor package deprecation, renaming, and redirecting added, in particular so that a package could be moved to npm by the author but could still be used as a Meteor package without needing to be copublished. For example some way to run a command or publish a final version of the Meteor package that would then map future versions to an npm package. This of course requires a solution to the question of how to identify client vs. server files if it’s a package that includes both. But even a solution that only works for the simple wrapper packages in the meantime would be nice. We can start weeding those out.

4 Likes
13

One possibility instead of using an imports folder could be to have a special file (package.json?) wherein modules can be specified using globs (similar to those in a .gitignore). F.e.:

{
  "modules": [
    "**/imports/**", // match all files in all imports folder.
  ]
}

This might be better than a modules or imports folder because some packages (semantic:ui for example) may rely on folders so named.

1 Like
14

If you guys do something like in the idea of my previous message – being able to specify modules with globs – then the guide could be structured in any arbitrary way with templates (etc) living next to module files. For example, suppose we want to use entirely ES2015 modules in a project, then:

{
  "modules": [
    "**/*.js", // match all js files anywhere.
  ]
}

Now we can place other files (css, html, etc) anywhere alongside the js files.

2 Likes
15

This is very good. So, we can put the current version of the app inside this folder and implement the new code in with modules. After that, we can slowly migrate other parts of the app while keeping the fast face of development.
(Basically, I don’t wanna spend 2months to convert the app to the new module system)

10 Likes
16

Any idea when Meteor 1.3 will be ready? Like more, or less than a month?

This will be very useful, particularly because there are so many ReactJS components on NPM, but when I try to bring them in, i.e. wrap them into meteor packages, something always goes wrong :frowning:

17

Probably a bit more than a month - sometime very early 2016. The team working on the module project has some of the initial functionality working and hopefully we can have an early pre-release in one or two weeks!

And yes, not having to wrap NPM packages in sketchy ways is one of the main benefits!

8 Likes
18

Wouldn’t NPM packages open Meteor to all kinds of databases? Even thought it wouldn’t be real time, it would be sweet.

2 Likes
19

Yeah the benefits are uncountable - React components, utility packages, database drivers, etc etc. Also, this can open the door to making some of the parts of Meteor more easily usable elsewhere - for example, the official ddp-client, tracker, and many others!

7 Likes
20

That’s going to be crack! :panda_face:

5 Likes
21

@sashko one thing I always wanted to ask. Since Blaze 2 will have React underneath, can I not simply write Vanilla React ?