@pbeato Seems that on Meteor@3.0 packages are loaded in parallel when possible, so that’s probably why you are getting a different order. You can read more about it here: https://github.com/meteor/meteor/blob/release-3.0/docs/source/api/top-level-await.md#module-and-package-execution-order.
If you want to load package A AFTER package B, you should list dependency B on the package.js A - in your case package three would depend on two and two would depend on one.
Thank you @matheusccastro, I understand the motivation behind this, but some packages, for example in redis-oplog, have based their operation precisely on the ability to have a predetermined loading order, this in order to be able to redesign a new arrangement of the base libraries before the next modules are executed. In this particular case, the early loading of the redis-oplog package allowed it to register all the collections to be monitored. If this is not possible, we would have to find another system, but if there is no order, I think it is complex to do without intervening in the kernel. We can’t even assume that every package that can mongo will be able to specify the redis-oplog library as optional. Redid-oplog is just one example, other packages use this stratagem to customise the way Meteor works.
Tagging @zodern since he was the one who implemented TLA for meteor
This is the direct text from the TLA docs
Module and Package Execution Order
Normally, modules are run one at a time. This was even true when using async code with fibers in the root of a module. However, top level await is different - it allows siblings (modules that do not depend on each other) to sometimes run in parallel. This can allow the app to load faster, which is especially important on the client. However, this could cause code to run in an unexpected order if you are used to how Meteor worked with fibers.
This is also applies to packages. Packages that do not directly or indirectly depend on each other are able to load in parallel if they use top level await.
Modules that are eagerly evaluated (added in packages with
api.addFiles, or outside ofimportsin apps that do not have a main module) and not directly imported continue to run one at a time, even if they use top level await, since it is common for these modules to implicitly depend on the previous modules.
Meteor 3 experiments with loading packages in parallel. I wasn’t sure if it would work or cause too many problems, but it’s easier (less of a breaking change) to reduce parallelism than to increase it if we made the wrong choice.
How it works is:
Most of the problems we’ve seen so far have been due to packages using properties on the Meteor object without having a dependency on the package that added those properties.
Do you know if redis oplog uses top level await? If it does not, then what you describe sounds like a bug since it should load synchronously before other packages that use mongo. If it does use top level await, then moving the code that needs to run before other packages to before the TLA should fix it.
Now that more apps are migrating to Meteor 3, it might be good to implement the serial loading so we can do some real world tests on if server start time is any better with parallel loading. If it’s not, then there would be no reason to keep it.
Thank you @zodern, if by asynchronous package you mean having used top-level await, I don’t recall ever having used that, is there any way to be sure?
The way some packages are designed now, switching to an asynchronous loading version definitely leads to problems that are not easily solved without modifying the core. Monky patches have often been used, perhaps abused, to extend the operation of Meteor, which has a predetermined order of loading. In some scenarios, there can be up to 5 levels of overloading of the collection class constructor. It is not always possible to use dependency logic, since each package should know which others are modifying a given feature. To have more control, we could think about including simple package logic lai:collection-extensions by the kernel; this would avoid some of the patches.
In a follow-up request, I report the following. I have modified the above example to simulate the redis-oplog package. I only changed the api.use clause of one package as follows:
api.use([ 'underscore
'underscore',
'ecmascript',
'ejson',
'minimongo',
'mongo',
'random',
'ddp-server',
'diff-sequence',
'id-map',
'mongo-id',
'tracker',
])
The order of the package files is
one
two
three
The result is
I20240117-11:02:43.688(1)? two.js
I20240117-11:02:43.689(1)? three.js
I20240117-11:02:43.689(1)? one.js
I20240117-11:02:43.691(1)? main
As you can see, the order has changed while apparently having no use for the top-level await.
Could you please create a reproduction I can try?
Hello @zodern, have you had a chance to verify the issue? Is this a behavior that will change in future versions?
Thank you.
Hello @fredmaiaarantes and @zodern, is there any news about this? For now, I have a custom version of core-runtime that forces sequential loading, but I don’t know if that is the best solution.
load-js-image.js
+ var names = []; // List of packages in order of loading
function queue(name, deps, runImage) {
pending[name] = [];
var pendingDepsCount = 0;
function onDepLoaded() {
pendingDepsCount -= 1;
if (pendingDepsCount === 0) {
load(name, runImage);
}
}
+ deps = [...new Set([...deps, ...names])]; // Add previous packages ad dependencies
deps.forEach(function (dep) {
if (hasOwn.call(pending, dep)) {
pendingDepsCount += 1;
pending[dep].push(onDepLoaded);
} else {
// load must always be called for a package's dependencies first.
// If the package is not pending, then it must have already loaded
// or is a weak dependency, and the dependency is not being used.
}
});
+ names.push(name); // Add name to list of packages
if (pendingDepsCount === 0) {
load(name, runImage);
}
}
Let me know.
A reply would be appreciated, it would help us to define our migration strategy. Thank you.
So many Meteor leaders copied in this thread and not a single one replies in 2 months …
Hey @pbeato, I’ll try to reach Zodern directly to see if they can help here.
I don’t have much knowledge of this part of the code, but if they’re not able to help, I can dedicate some time to try to understand this and fix it.
Sorry for not replying sooner. I haven’t had much free time for open source work lately.
I looked into your reproduction. Packages two and three load in the correct order after the mongo package loads. Package one loads later since it also has to wait for the ddp-server package to load. It’s working by design.
We probably do need to switch back to loading packages one at a time. @denyhs, if you want to fix it, it mostly involves removing all of the deps code from core runtime and the linker. Packages are already queued in the core runtime in the order they should be loaded. Otherwise, I might have time later this month.
Does switching back mean slower load times?
Possibly. The goal was to improve loading times, but I don’t think we have any data on if how much of a difference it makes. This is probably more important for the client to reduce the effect on load time if multiple packages use top level await, but TLA on the client is currently disabled by default and it doesn’t seem that will change before Meteor 3.
One important aspect of Meteor’s design for top level await is most packages can start or stop using top level await without it being a breaking change (two exceptions are packages with lazy main modules, and when specific files are imported from the package). However, if a package using top level await changes the load order of other packages enough that they could break, then that is no longer true. I’m not sure there is a way to fix this without loading packages one at a time.
FYI, I’ve opened this PR to solve this issue. I’m getting an error, as described in the description, but hopefully, we’ll be able to figure this out soon 