I am porting my application to Meteor 3.0. Compared to the 2.x version, I am noticing a different loading order of packages. I noticed that in version 3.0 you get a different loading order when you put a dependency on mongo.
I have a simple application with three packages in it: one, two and three. For each of them I just do a console.log of the filename. When I add the reference to mongo, api.use(‘mongo’), in the first and third packages.js, I get a different order of loading the packages. This order does not respect the packages of the project.
meteor-base@1.5.1
one
two
three
shell-server@0.5.0
dynamic-import@0.7.3
underscore@1.0.13
mongo@1.16.8
When I launch this application with version 2.14 I get:
=> Started proxy.
=> Started MongoDB.
I20240115-17:48:07.472(1)? one.js
I20240115-17:48:07.489(1)? two.js
I20240115-17:48:07.490(1)? three.js
=> Started your app.
With version 3.0 I get:
=> Started proxy.
=> Started MongoDB.
I20240115-17:47:24.100(1)? two.js
I20240115-17:47:24.248(1)? one.js
I20240115-17:47:24.248(1)? three.js
=> Started your app.
This variation in sorting in my case generates many problems. Is there anything I can do to keep the sequence correct?
If you want to load package AAFTERpackage B, you should list dependency B on the package.js A - in your case package three would depend on two and two would depend on one.
Thank you @matheusccastro, I understand the motivation behind this, but some packages, for example in redis-oplog, have based their operation precisely on the ability to have a predetermined loading order, this in order to be able to redesign a new arrangement of the base libraries before the next modules are executed. In this particular case, the early loading of the redis-oplog package allowed it to register all the collections to be monitored. If this is not possible, we would have to find another system, but if there is no order, I think it is complex to do without intervening in the kernel. We can’t even assume that every package that can mongo will be able to specify the redis-oplog library as optional. Redid-oplog is just one example, other packages use this stratagem to customise the way Meteor works.
Tagging @zodern since he was the one who implemented TLA for meteor
This is the direct text from the TLA docs
Module and Package Execution Order
Normally, modules are run one at a time. This was even true when using async code with fibers in the root of a module. However, top level await is different - it allows siblings (modules that do not depend on each other) to sometimes run in parallel. This can allow the app to load faster, which is especially important on the client. However, this could cause code to run in an unexpected order if you are used to how Meteor worked with fibers.
This is also applies to packages. Packages that do not directly or indirectly depend on each other are able to load in parallel if they use top level await.
Modules that are eagerly evaluated (added in packages with api.addFiles, or outside of imports in apps that do not have a main module) and not directly imported continue to run one at a time, even if they use top level await, since it is common for these modules to implicitly depend on the previous modules.
Meteor 3 experiments with loading packages in parallel. I wasn’t sure if it would work or cause too many problems, but it’s easier (less of a breaking change) to reduce parallelism than to increase it if we made the wrong choice.
How it works is:
it starts by loading packages in order, one at a time
when a package uses top level await, the packages that depend on it wait until the package finishes loading, and then resume loading one at a time
other packages that don’t depend on it load in the meantime
Most of the problems we’ve seen so far have been due to packages using properties on the Meteor object without having a dependency on the package that added those properties.
Do you know if redis oplog uses top level await? If it does not, then what you describe sounds like a bug since it should load synchronously before other packages that use mongo. If it does use top level await, then moving the code that needs to run before other packages to before the TLA should fix it.
Now that more apps are migrating to Meteor 3, it might be good to implement the serial loading so we can do some real world tests on if server start time is any better with parallel loading. If it’s not, then there would be no reason to keep it.
Thank you @zodern, if by asynchronous package you mean having used top-level await, I don’t recall ever having used that, is there any way to be sure?
The way some packages are designed now, switching to an asynchronous loading version definitely leads to problems that are not easily solved without modifying the core. Monky patches have often been used, perhaps abused, to extend the operation of Meteor, which has a predetermined order of loading. In some scenarios, there can be up to 5 levels of overloading of the collection class constructor. It is not always possible to use dependency logic, since each package should know which others are modifying a given feature. To have more control, we could think about including simple package logic lai:collection-extensions by the kernel; this would avoid some of the patches.
In a follow-up request, I report the following. I have modified the above example to simulate the redis-oplog package. I only changed the api.use clause of one package as follows:
Hello @fredmaiaarantes and @zodern, is there any news about this? For now, I have a custom version of core-runtime that forces sequential loading, but I don’t know if that is the best solution.
load-js-image.js
+ var names = []; // List of packages in order of loading
function queue(name, deps, runImage) {
pending[name] = [];
var pendingDepsCount = 0;
function onDepLoaded() {
pendingDepsCount -= 1;
if (pendingDepsCount === 0) {
load(name, runImage);
}
}
+ deps = [...new Set([...deps, ...names])]; // Add previous packages ad dependencies
deps.forEach(function (dep) {
if (hasOwn.call(pending, dep)) {
pendingDepsCount += 1;
pending[dep].push(onDepLoaded);
} else {
// load must always be called for a package's dependencies first.
// If the package is not pending, then it must have already loaded
// or is a weak dependency, and the dependency is not being used.
}
});
+ names.push(name); // Add name to list of packages
if (pendingDepsCount === 0) {
load(name, runImage);
}
}
Sorry for not replying sooner. I haven’t had much free time for open source work lately.
I looked into your reproduction. Packages two and three load in the correct order after the mongo package loads. Package one loads later since it also has to wait for the ddp-server package to load. It’s working by design.
We probably do need to switch back to loading packages one at a time. @denyhs, if you want to fix it, it mostly involves removing all of the deps code from core runtime and the linker. Packages are already queued in the core runtime in the order they should be loaded. Otherwise, I might have time later this month.
Possibly. The goal was to improve loading times, but I don’t think we have any data on if how much of a difference it makes. This is probably more important for the client to reduce the effect on load time if multiple packages use top level await, but TLA on the client is currently disabled by default and it doesn’t seem that will change before Meteor 3.
One important aspect of Meteor’s design for top level await is most packages can start or stop using top level await without it being a breaking change (two exceptions are packages with lazy main modules, and when specific files are imported from the package). However, if a package using top level await changes the load order of other packages enough that they could break, then that is no longer true. I’m not sure there is a way to fix this without loading packages one at a time.
FYI, I’ve opened this PR to solve this issue. I’m getting an error, as described in the description, but hopefully, we’ll be able to figure this out soon
Just an update here. @radekmie was able to help us here and created this PR which solves the loading order. There is a testing “failing” now, so we’re looking into that.
