Order of loading packages

I am porting my application to Meteor 3.0. Compared to the 2.x version, I am noticing a different loading order of packages. I noticed that in version 3.0 you get a different loading order when you put a dependency on mongo.

I have a simple application with three packages in it: one, two and three. For each of them I just do a console.log of the filename. When I add the reference to mongo, api.use(‘mongo’), in the first and third packages.js, I get a different order of loading the packages. This order does not respect the packages of the project.

Package one:

Package.onUse(function(api) {
  api.versionsFrom(['2.14','3.0-beta.0']);
  api.use('ecmascript');
  api.use('mongo');
  api.addFiles('one.js');
});

Package two:

Package.onUse(function(api) {
  api.versionsFrom(['2.14','3.0-beta.0']);
  api.use('ecmascript');
  // api.use('mongo');
  api.addFiles('two.js');
});

Package three:

Package.onUse(function(api) {
  api.versionsFrom(['2.14','3.0-beta.0']);
  api.use('ecmascript');
  api.use('mongo');
  api.addFiles('three.js');
});

File .meteor/packages

meteor-base@1.5.1
one
two
three
shell-server@0.5.0
dynamic-import@0.7.3
underscore@1.0.13
mongo@1.16.8

When I launch this application with version 2.14 I get:

=> Started proxy.                             
=> Started MongoDB.                           
I20240115-17:48:07.472(1)? one.js             
I20240115-17:48:07.489(1)? two.js
I20240115-17:48:07.490(1)? three.js
=> Started your app.

With version 3.0 I get:

=> Started proxy.                             
=> Started MongoDB.                           
I20240115-17:47:24.100(1)? two.js             
I20240115-17:47:24.248(1)? one.js
I20240115-17:47:24.248(1)? three.js
=> Started your app.

This variation in sorting in my case generates many problems. Is there anything I can do to keep the sequence correct?

@pbeato Seems that on Meteor@3.0 packages are loaded in parallel when possible, so that’s probably why you are getting a different order. You can read more about it here: https://github.com/meteor/meteor/blob/release-3.0/docs/source/api/top-level-await.md#module-and-package-execution-order.

If you want to load package A AFTER package B, you should list dependency B on the package.js A - in your case package three would depend on two and two would depend on one.

Thank you @matheusccastro, I understand the motivation behind this, but some packages, for example in redis-oplog, have based their operation precisely on the ability to have a predetermined loading order, this in order to be able to redesign a new arrangement of the base libraries before the next modules are executed. In this particular case, the early loading of the redis-oplog package allowed it to register all the collections to be monitored. If this is not possible, we would have to find another system, but if there is no order, I think it is complex to do without intervening in the kernel. We can’t even assume that every package that can mongo will be able to specify the redis-oplog library as optional. Redid-oplog is just one example, other packages use this stratagem to customise the way Meteor works.

Tagging @zodern since he was the one who implemented TLA for meteor

This is the direct text from the TLA docs

Module and Package Execution Order

Normally, modules are run one at a time. This was even true when using async code with fibers in the root of a module. However, top level await is different - it allows siblings (modules that do not depend on each other) to sometimes run in parallel. This can allow the app to load faster, which is especially important on the client. However, this could cause code to run in an unexpected order if you are used to how Meteor worked with fibers.

This is also applies to packages. Packages that do not directly or indirectly depend on each other are able to load in parallel if they use top level await.

Modules that are eagerly evaluated (added in packages with api.addFiles, or outside of imports in apps that do not have a main module) and not directly imported continue to run one at a time, even if they use top level await, since it is common for these modules to implicitly depend on the previous modules.

Meteor 3 experiments with loading packages in parallel. I wasn’t sure if it would work or cause too many problems, but it’s easier (less of a breaking change) to reduce parallelism than to increase it if we made the wrong choice.

How it works is:

  • it starts by loading packages in order, one at a time
  • when a package uses top level await, the packages that depend on it wait until the package finishes loading, and then resume loading one at a time
  • other packages that don’t depend on it load in the meantime

Most of the problems we’ve seen so far have been due to packages using properties on the Meteor object without having a dependency on the package that added those properties.

Do you know if redis oplog uses top level await? If it does not, then what you describe sounds like a bug since it should load synchronously before other packages that use mongo. If it does use top level await, then moving the code that needs to run before other packages to before the TLA should fix it.

Now that more apps are migrating to Meteor 3, it might be good to implement the serial loading so we can do some real world tests on if server start time is any better with parallel loading. If it’s not, then there would be no reason to keep it.

Thank you @zodern, if by asynchronous package you mean having used top-level await, I don’t recall ever having used that, is there any way to be sure?

The way some packages are designed now, switching to an asynchronous loading version definitely leads to problems that are not easily solved without modifying the core. Monky patches have often been used, perhaps abused, to extend the operation of Meteor, which has a predetermined order of loading. In some scenarios, there can be up to 5 levels of overloading of the collection class constructor. It is not always possible to use dependency logic, since each package should know which others are modifying a given feature. To have more control, we could think about including simple package logic lai:collection-extensions by the kernel; this would avoid some of the patches.

In a follow-up request, I report the following. I have modified the above example to simulate the redis-oplog package. I only changed the api.use clause of one package as follows:

api.use([ 'underscore
    'underscore',
    'ecmascript',
    'ejson',
    'minimongo',
    'mongo',
    'random',
    'ddp-server',
    'diff-sequence',
    'id-map',
    'mongo-id',
    'tracker',
  ])

The order of the package files is

one
two
three

The result is

I20240117-11:02:43.688(1)? two.js
I20240117-11:02:43.689(1)? three.js
I20240117-11:02:43.689(1)? one.js
I20240117-11:02:43.691(1)? main

As you can see, the order has changed while apparently having no use for the top-level await.

Could you please create a reproduction I can try?

I created a simple project that reproduces the problem.

Hello @zodern, have you had a chance to verify the issue? Is this a behavior that will change in future versions?

Thank you.

Hi @zodern is there any news about this? Thanks