Slow Meteor Package Extraction

help
1

Perhaps foolish, but I have setup Docker Desktop on Windows (with WSL2). I have a Dockerfile with an Ubuntu Jammy base to install nvm, npm, meteor, and other tools I need for my project.

However, startup is rather slow - in the order of 10-15 minutes. After profiling, it looks like the package download / extraction phase is where things get slow:

On meteor startup, I see a large number of Downloading missing local versions of package messages, e.g.

Downloading missing local versions of package wildhart:jobs@1.0.13 : [
'os.linux.x86_64' ]
Doing HTTP request:  GET
https://warehouse.meteor.com/builds/zguQ3XdNWQypSnBfH/1629659446009/tE2QvAtfoN/wildhart:jobs-1.0.13-os+web.browser+web.browser.legacy+web.cordova.tgz

This is followed by the Finished extracting in 349 ms messages (I assume extracting the above tarballs)

Finished extracting in 349 ms
Finished extracting in 563 ms
Finished extracting in 753 ms
Finished extracting in 560 ms
Finished extracting in 979 ms
Finished extracting in 1061 ms
Finished extracting in 1098 ms
Finished extracting in 1171 ms
Finished extracting in 1205 ms
Finished extracting in 905 ms
Finished extracting in 1296 ms
Finished extracting in 1352 ms
Finished extracting in 1520 ms
Finished extracting in 1445 ms
Finished extracting in 1654 ms
Finished extracting in 1828 ms
Finished extracting in 1988 ms
Finished extracting in 2204 ms
Finished extracting in 2193 ms
Finished extracting in 3046 ms
Finished extracting in 3654 ms
Finished extracting in 3182 ms
Finished extracting in 3998 ms
Finished extracting in 4351 ms
Finished extracting in 4710 ms
Finished extracting in 4758 ms
Finished extracting in 5063 ms
Finished extracting in 5112 ms
Finished extracting in 4762 ms
Finished extracting in 5673 ms
Finished extracting in 6041 ms
Finished extracting in 6904 ms
Finished extracting in 10875 ms
Finished extracting in 11831 ms
Finished extracting in 27532 ms
Finished extracting in 31273 ms
Finished extracting in 53387 ms
Finished extracting in 157720 ms
Finished extracting in 206713 ms
Finished extracting in 462949 ms
Finished extracting in 516493 ms
Finished extracting in 566498 ms
Finished extracting in 602415 ms

Note: 602415 ms = 10.04025 minutes

I was hoping to:

  1. Find the directory to which these files are extracted
  2. Add the directory as a Docker Compose volume to avoid this process on startup
2

You should either link in the meteor directory so these things live outside docker, or you should add just your meteor packages file then run a meteor command before adding all your app files, to make use of Dockers built in layer cache.

The former method will be much more effective as the layer cache would be invalidated every time you add a package, requiring re-download of every package.

This probably isn’t a meteor problem, but if you share your docker file (and docker compose file or commands) we can probably help more.

3

Thanks for the reply! I am similarly leaning towards volume-izing the folder in which these packages live, but wasn’t sure where that was exactly and so here I am…

You should either link in the meteor directory

Which directory would that be:

  • ~/.meteor
  • ~/project-name/.meteor

Guessing the former here?

This probably isn’t a meteor problem

I broadly agree with this statement, the performance issue itself is with WSL2 insofar as I am aware, though there are potential improvements to the debug logging I think could fall out of my pursuit.

  1. Changing Finished extracting in N ms to Finished extracting $PACKAGE in N ms to $TARGET
  2. Adding a debug message indicating the package used to extract the tarballs.

Regarding #2 - I’ve found some many-years-old posts indicating that Meteor’s strategy is the look for libraries on the path to unzip, and then falling back to a pure-JS implementation. It would be good to know from scanning the logs what is going on under the hood.