Improve build and reload times for a complex app structure

help
1

I am a member of a team and we are working on several apps, which consist in shared meteor packages among the apps. We have had problems on the load/reload times of the apps and packages tests over the time, and is becoming painful as the project grows. In this moment we can consume more than 2 minutes on app first load, on subsequent loads much less because of the meteor caching strategy. Still it can get 30-50 seconds to reload on package code change. This is hitting the team’s productivity and happiness, and we need desperately solutions for it.

Project infrastructure

First, I want you to understand how our apps and packages are shaped.

Meteor App #1
   Meteor Package A
      NPM devDependencies
   Meteor Package B
      NPM devDependencies
Meteor App #2
   Meteor Package A
      NPM devDependencies
  • Each Meteor App has Meteor Packages attached
  • Meteor Packages are managed in an external folder (not in /packages) as are shared between Meteor Apps
  • Meteor Packages include their own devDependencies for testing and editor purposes (babel, chai, eslint, etc)
  • For running tests we do it from package level and not app level (as packages are shared)

Possible issues to improve the performance

There are probably different stuff we should consider, most of them are described here and we will try them out. However I was able to notice an issue that could affect us significatively and is not reported there. I reproduced such in a small-sized repository.

Reproduction: Meteor packages describing devDependencies

The structure of the repository is:

Meteor App #1
   Meteor Package A
  • Meteor App #1 includes Meteor Package A
  • Meteor Package A doesn’t include devDependencies

Then I tried the next stuff and I got the next the timing results:

  1. Run the app
prepareProjectForBuild - Total: 3,217 ms
Rebuild App - Total: 193 ms
  1. Run the app again
prepareProjectForBuild - Total: 74 ms
Rebuild App - Total: 172 ms
  1. Code change on the package
prepareProjectForBuild - Total: 235 ms
Rebuild App - Total: 221 ms
  1. Include the next devDependencies on the package and run the app
"devDependencies": {
    "babel-eslint": "^6.1.2",
    "babel-plugin-transform-class-properties": "^6.16.0",
    "babel-preset-es2015": "^6.16.0",
    "babel-preset-stage-0": "^6.16.0",
    "eslint": "^2.9.0",
    "eslint-config-airbnb": "^6.2.0",
    "eslint-plugin-babel": "^3.3.0",
    "eslint-plugin-import": "^1.6.1",
    "eslint-plugin-jsx-a11y": "^1.0.4",
    "eslint-plugin-react": "^4.3.0",
    "jsdom": "9.12.0",
    "chai": "^3.5.0",
    "chai-datetime": "^1.4.1",
    "cheerio": "^0.20.0",
    "sinon": "^1.17.4",
    "sinon-chai": "^2.8.0"
  }

prepareProjectForBuild - Total: 82,815 ms
Rebuild App - Total: 33,231 ms
  1. Run the app again
prepareProjectForBuild - Total: 25,997 ms
Rebuild App - Total: 3,822 ms
  1. Change code on the package
prepareProjectForBuild - Total: 133,085 ms
Rebuild App - Total: 744 ms

Analysis

  • The scenario of running a simple app with a package attached (1) takes the less time to build and a continuous run (2) or change on a file (3) cause the timing to be faster as the meteor cache acts.

  • As we add dependencies on the package, both the time required for prepare and build the app increase significatively with the amount of npm modules installed (4), even if these are devDependencies! Again the cache acts on continuos runnings (5), but is even worse on change (6).

As you can see in this small reproduction this could explain the amount of time that our project takes to build as the project has a lot of packages describing devDependencies modules.

Possible solutions and questions

We may are wrong on our infrastructure of apps and shared packages, and you may have some suggestions on this. However, I can think in two solutions and questions about the impact on those.

  1. Remove all devDependencies from all packages. But,

    • What would happen on run packages tests? Should these then depend on a separate meteor app that runs them?
    • As we remove eslint dependency, our editor could not detect the linter when working on the packages. The same would happen with all dev dependencies that support editor features. How do we solve this?

  2. Use the recently introduced .meteorignore feature on all packages to exclude all devDependencies. This will solve the editor issues. But,

    • What would happen on run packages tests?
2

Packages look for the top level projects npm dependencies and there’s even a great package to check the presence of the dependencies from within your package code.

So, based on this piece of information, I’d advise you to create three apps:

  1. A package-host-app with no app code in it, only dependencies and devDependencies. packages folder and all your packages reside here. this is where you do package development and testing. No dependencies or devDependencies should exist within individual packages, in fact don’t even use package.json files in there. Just keep the one in the top level.

  2. App1 with no packages folder, but starts up with METEOR_PACKAGE_DIRS="/path/to/package-host-app/packages/" env variable set. App1 has only the dependencies relevant to the packages added to this app installed. this is where you develop app 1 where you meteor add your:packages relevant to App1

  3. App2 with no packages folder, but starts up with METEOR_PACKAGE_DIRS="/path/to/package-host-app/packages/" env variable set. App1 has only the dependencies relevant to the packages added to this app installed. this is where you develop app 2 where you meteor add your:packages relevant to App2

  4. Rinse and repeat

If you want, you can also create a packages folder in any one of your apps to create packages that are exclusively dedicated to that app.

Note: instead of the environment variable, you may also create a symlink named packages within your apps and point it to the packages folder of the host app, but this might be confusing to your ide/editor in cases where the application is not using all the packages. IDE’s tend to be over aggressive trying to provide autocompletion.

For best autocompletion, you might want to create some typescript typings for your package apis (after all the packages ought to expose small api surfaces) and utilize these for code assistance.

2 Likes
3

Thanks a lot for your reply. It has been really helpful.

We have moved to the approach that you proposed last week and we have gained many benefits, not only in terms of performance, but also for managing the modules easier.

Packages look for the top level projects npm dependencies and there’s even a great package to check the presence of the dependencies1 from within your package code.

This was something we tried on the past, but we couldn’t not get the benefits we wanted. For handling peerDependencies between our apps and packages, we just created a new meteor packages where we hold all dependencies with its proper version. Then we reference them on imports like import { identity } from 'meteor/vendor/lodash'.

We have got the benefits to support peer dependencies and a single point to handle them and to improve the performance on build, since the app has not to handle all deps at time. For large projects with a lot of dependencies we have noted that is better to keep and expose them on a different package.

1 Like
4

Well this kind of dependency “rerouting” is a common practice from another perspective, too, where you re-export the parts of the dependency - sometimes in a custom namespace of your own so that it becomes an abstraction. This way, you can simply switch your dependency to another library (maintaining the signatures exposed by your exports) without needing to refactor the app codebase.

This is something I quite commonly do for utility libraries like lodash, moment, some string and number libraries. This approach has already enabled me to seamlessly switch away from underscore, moment, underscore.string and a few different number/money libraries I used in the past to either their native implementations or alternative libraries.

2 Likes