Improved build server caching, e.g. using --prepare-app?

performance
1

We have used the great tips provided by @zodern in this blog post for many years now on circleci, but are looking for more good practices to improve build performance.

The time to download and install 3rd party meteor packages is not insignificant and we stumbled upon the “internal” --prepare-app command which does just that.

The idea would be to cache the entire $HOME/.meteor folder with a cache key of meteor-{{ checksum “.meteor/versions” }} after populating it with --prepare-app.

Is anyone already doing this? Do you also save away the modifications to the PATH variable that meteor install adds?

In any case, I will report back when we have tested this method.

1 Like
2

Initial testing looks promising - it shaved about one minute off my build times AND made the builds independent of the flakiness troubles we’ve seen recently.

Here is the complete circlei command that we use (copy the parts that apply to you):

commands:
  setup:
    steps:
      - checkout
      - restore_cache:
          name: Restore Yarn v2 Cache
          keys:
            - yarn2-cache-{{ checksum ".meteor/versions" }}-{{ checksum "yarn.lock" }}
            - yarn2-cache-{{ checksum ".meteor/versions" }}-
      - restore_cache:
          name: Restore .meteor/local subset
          keys:
            - meteor-local-{{ checksum ".meteor/versions" }}-{{ .Branch }}-{{ .Revision }}
            - meteor-local-{{ checksum ".meteor/versions" }}-{{ .Branch }}-
            - meteor-local-{{ checksum ".meteor/versions" }}-
      - restore_cache:
          name: Restore global meteor installation
          keys:
            - meteor-global-{{ checksum ".meteor/release" }}-{{ checksum ".meteor/versions" }}
      - run:
          name: Test if meteor is installed and add symlink (commands copied from meteor install script)
          command: |
            if [ -d "$HOME/.meteor" ]; then
              METEOR_SYMLINK_TARGET="$(readlink "$HOME/.meteor/meteor")"
              METEOR_TOOL_DIRECTORY="$(dirname "$METEOR_SYMLINK_TARGET")"
              LAUNCHER="$HOME/.meteor/$METEOR_TOOL_DIRECTORY/scripts/admin/launch-meteor"
              PREFIX="/usr/local"
              echo "Copying meteor launch script from '$LAUNCHER' to '$PREFIX/bin/meteor'"
              sudo cp "$LAUNCHER" "$PREFIX/bin/meteor" 
              which meteor
            fi
      - run:
          name: Install meteor if needed
          command: |
            export METEOR_VERSION=$(awk '{split($0, a, "@"); print a[2]}'  < .meteor/release)
            which meteor || timeout 45s bash -c '(curl -s "https://install.meteor.com/?release=${METEOR_VERSION}" | /bin/sh)'
          no_output_timeout: 1m # Sometimes meteor does not download, crash early.
      - run:
          name: Downloading meteor package dependencies specified in .meteor/versions via .meteor/packages
          command: |
            meteor --prepare-app
      - save_cache:
          name: Saving meteor global installation
          key: meteor-global-{{ checksum ".meteor/release" }}-{{ checksum ".meteor/versions" }}
          paths:
            - "~/.meteor"

our other save_cache steps look like this:

      - save_cache:
          name: Saving yarn 2 cache from ./.yarn/cache
          key: yarn2-cache-{{ checksum ".meteor/versions" }}-{{ checksum "yarn.lock" }}
          paths:
            - ./.yarn/cache


      - save_cache:
          name: Saving meteor cache files
          key: meteor-local-{{ checksum ".meteor/versions" }}-{{ .Branch }}-{{ .Revision }}
          paths: # https://zodern.me/posts/meteor-local-folder/#caching-in-ci
            - "./.meteor/local/isopacks"
            - "./.meteor/local/bundler-cache/scanner"
            - "./.meteor/local/plugin-cache"
2 Likes