A possible solution to the problem of incremental loading

core
1

Hi there!


It’s been quite a long time and still there’s no satisfactory answer to the following question:

How to split a meteor application into chunks and deliver them to the client on demand?

It could be quite useful to optimize the application loading time in scenarios when you want to serve a slightly different subset of your source code depending on the user’s role.

Current situation

Looking at the Meteor roadmap here:

it seems there’s been a lot of experiments around this subject, most of them focusing on the area of a source code delivery mechanism. Though, we all know that the real problem here is not in the code delivery, but in some obstructions in the project Meteor build mechanism. Namely, there’s currently no way to tell the “bundler” to split your client source code into several chunks, and there’s also no way to ignore certain files during the build process.

We’ve been hearing a lot answers of it-is-by-design type, and I am pretty much sure I can understand the reasoning behind that. I also believe that there are a lot of developers who knows well what they’re doing and they could use this “chunks feature” in a good way.

Some time ago, I’ve done my own research in the topic:


but it was while the big changes were happening to the Meteor packaging system so it was not realistic to encourage MDG to accept any additional development in that area.

Possible solution?

I think that we are currently in a situation when we can at least discuss some possible solutions to the problem I described above.

One idea I’ve been thinking about was to have a file, e.g.

.meteor/chunks, .meteor/client_bundles or just .meteor/bundles

describing which files should be included with particular chunks during the build process

[default]
  include client/**/*
  exclude client/dashboard/**/*

[dashboard]
  include client/dashboard/**/*

Then in the __meteor_runtime_config__ we may have a dictionary mapping a chunk name to a list of corresponding “bundled” files. From know on, it only depends on the developer how those files are loaded. Of course the default bundle should be included in <head> tag the same way it’s done right now.

Question

Please let me know what do you guys think about this and if it’s realistic to finally get this feature implemented. I can always try to take responsibility for the “hard work” but I don’t really want to waste time if the other developers and MDG thinks it’s a bad idea. So I am waiting for your feedback guys.

1 Like
2

How would the client know/decide when to load which chunk?

3

@serkandurusoy Good question. But the answer here is just “it does not matter” because it reaches beyond the scope of this project. I imagine that based on these features one might easily build an iron-router integration package, where the files could be loaded based on the current route for example.

Anyway, there are a lot of options to go once we have chunks, but we can’t have them until some updates are done to the build tool, which is what this post is about.

1 Like
4

Associate routes with packages, dependencies and collections. That drastically reduces lot of overhead.

Set the dependent packages defined per route/page with backward compatibility .

Something like,

api.use({...}, "client", route_id) // only for that route
api.use({...}, "client") // global and loaded for all
1 Like
5

I tend to disagree. Routes are certainly related to what’s needed but coupling them like this makes the api.use() list too verbose and redundant. Many packages will see use to lots of routes and that may soon become unmanageable.

7

:+1: On-demand template loading would address security concerns (rendering admin templates to any client) and performance concerns (sending all templates on the wire when only some are used most of the time).

Just thought I’d add some prior art:

8

Good luck working on this. My publicsources and privatesources that @dandv mentioned provide a way to load incrementally but do leave a desire for an easier solution.

Very similarly to your .meteor/bundle file, I have thought a little bit about a .meteor/dirmap file that would default to loading files as Meteor currently does but could be customized to make it so that Meteor would work without locking the developer into a specific directory structure. If source files could specify their dependencies with something like Meteor.addFiles(), an incremental loader would be able to determine which files are required for specific templates.