I’m trying to improve the build time of docker images for meteor apps,
When building meteor apps for production, it seems that most (potentially all?) server side node modules get some combination of .meteor-portable.json and .meteor-portable-2.json - most of which contain true but a handful of which contain false (protobufjs, core-js, resolve, nodemailer) - some of these files aren’t at the top level of the directory (build/bundle/programs/server/npm/node_modules/meteor/babel-compiler/node_modules/resolve/test/resolver/multirepo/.meteor-portable-2.json) and some are.
Similarly, some modules seem to have left behind “garbage” files .resolve-garbage-5k0w9p.r9qj - these are randomly generated.
I can’t find any information about these files - though the changelog alludes to “non-portable node modules being rebuilt”
Does anyone know:
What is the difference between .meteor-portable.json and .meteor-portable-2.json
Why are these added at all
Do the values in these files ever change if the package version doesn’t change (this is what really matters as I’m trying to deal with caching)
What would happen if I were to remove all these files after the build, but prior to dockerizing? And follow up, what about only the false ones
When you deploy, Meteor bundles all of the app’s npm dependencies (except dev dependencies). Since many people deploy to a different arch than they are deploying from, any native dependencies will have to be rebuilt. Same with packages - they are published from one architecture but can be run on any so Meteor takes care of rebuilding native npm dependencies when the package is downloaded. It takes a few seconds to analyze a node_modules directory to find the native dependencies, so Meteor creates the json files to cache some of the work.
What is the difference between .meteor-portable.json and .meteor-portable-2.json
The initial implementation didn’t find all of the native dependencies. When it was fixed the file name was changed so it wouldn’t use the old files that could have inaccurate data.
Do the values in these files ever change if the package version doesn’t change (this is what really matters as I’m trying to deal with caching)
No. They only change when the package is modified (which is why they are in node_modules - they get automatically deleted when the package manager updates the package)
What would happen if I were to remove all these files after the build, but prior to dockerizing? And follow up, what about only the false ones
They are only used during the build. Meteor writes the results in the built app at programs/server/npm-rebuilds.json and uses that when npm install is run to rebuild the native dependencies.
Thanks for this! Any thoughts on why 2 identical builds (without any code changes) would have invalid caches based on the presence of these files (I can ensure a docker cache hit by removing these files before adding to docker)? And if .meteor-portable.json is deprecated, why they are still being added?
Similarly, when you say these files are cached - where are they cached? From what I can tell my build/bundle directory is created clean each time - is it pulling these files in from .meteor/local/build/programs/server/npm/node_modules/? I dont see these files there at all - I do see them in .meteor/local/isopacks/ - but that would only account for meteor packages node dependencies.
Lastly - any thoughts on why the *-garbage-* directories are kept? Those are the other things that cause problems with the docker cache.
Thanks for this! Any thoughts on why 2 identical builds (without any code changes) would have invalid caches based on the presence of these files (I can ensure a docker cache hit by removing these files before adding to docker)?
I’m not sure. I don’t remember Meteor doing anything that would cause this, but it has been a while since I looked at this part of the Meteor Tool. Does the file’s metadata change each time?
And if .meteor-portable.json is deprecated, why they are still being added?
Usually they are from Meteor packages published before the file name was changed.
Similarly, when you say these files are cached - where are they cached?
The files are the cache. Instead of examining the npm package, Meteor only needs to read the file.
Lastly - any thoughts on why the *-garbage-* directories are kept? Those are the other things that cause problems with the docker cache.
Could you please give me the full path of one of these garbage directories (or at least relative to the app or bundle)?