Build Speeds for Development are unreasonable, can we parallelize them or improve them further?

core
1

I’ve been experimenting with build times for Meteor and have come to the conclusion that for large or the smallest/blank newly created apps are about 20 seconds (a large production app with cached bundled assets takes 20-22 seconds, a blank application with cached bundled assets takes 17-20 seconds).

I believe the biggest reason for this is Meteor is 1.) not parallelizing and 2) doing to much.

On 1.) node and meteor are limited by the event loop so asset compilation happens in series, rather then utilizing multiple cores, this slows things down a lot. I’m not sure where the resistance is to compile each plugin in it’s own thread via a thread pool. I’m sure some things must be done in series when collecting assets.

On 2.) Doing to much, for one example after a build and exiting Meteor, the build process begins entirely again rather than determining if anything in the application has actually changed, so the whole process begins again (however starting node .meteor/local/build/main.js is nearly instant). Since Meteor is limited by the event loop and not IO, we should consider CPU the more available resource I believe. As example I tested on a machine w/e an NVMe that can read & write around 2GB/s, more than the entire assets of a project.

Additionally but less investigated Meteor minifies in development, or at the very least you can see something happening in when run with METEOR_PROFILE. Which seems odd.

Results were compared from a Macbook Retina & a Dell R820 (4*2.7GHz - 64 threads, 256GB, 2GB/s nvme SSD), results we’re comparable between both devices which both run at 2.7GHz, and boost to 3.3GHz.

Now, however for rebuild times we do an excellent job, and things can be only a 1-3 seconds, which is great!

With all this said, how can we improve meteors build process and where can a developer start. Because it feels like the build code is bit of a maze, some higher level documentation would be nice to have here.

Related Feature Request: Improve Build times by Utilizing webworkers/threads · Issue #312 · meteor/meteor-feature-requests · GitHub

6 Likes
2

It would be amazing to improve on that. For me, that’s by far the biggest downside of developing in Meteor. In my app, every code change takes ages. And it’s not that big.

3

Not every code change, reloads are very decent, but as part of the work flow how often you need to restart meteor adds up. 20 seconds. Considering an 8 hour working day, restarting meteor every 5 minutes, (860)/5 = 96 times, now multiply that unfairly by 20 seconds, (9620)/60=32 mins half hour a day, times that by 20 working days (32*20)/60=640mins (10.6 hours) a month, and now times that across my team of 5 developers, looking at up to 50 hours a month waiting for meteor to build. Which costs more than all our cloud bills combined and we’re not a small company anymore. If MDG is looking for revenue, it’s not in hosting, it’s in building tools that save me time, I’ll/we’ll pay for them.

I digress! Reloads are quick!

1 Like
4

I certainly don’t want to hinder progress on any front, but I have to say that for me personally build times on Meteor have never been an issue. Like you said, rebuild times are great (i.e. reacting to my own changes during development), only 1-3 seconds in my app as well, and it seems to me that this is what matters the most.

Building from scratch does indeed take more time (still less than a minute for me), but this is something I do much less often. Maybe once every hour or so, maybe once every couple of hours. And because of this, personally for me those extra seconds have little to no impact. Your workflow seems a lot different, considering a meteor restart every 5 minutes. I hope you don’t mind me asking, but what would cause a need to restart meteor so frequently?

I’m only asking since being a big proponent of the KISS principle I’d be wary of introducing any further complexity (and thereby bugs to hunt us in the future) to the build process unless there’s considerable benefits justifying the hit.

1 Like
5

Word. Once you factor in the trips to the bathroom, moments of stretching, and the longer-than-necessary lunch breaks, it gets really crazy.

1 Like
6

Life of luxury there. I barely even have time to code. /s

7

I think 1.7 has an issue right now where it’s doing legacy browser builds during development. They’re adding a flag to turn that off.

If you’re not using 1.7, then I’m not sure.

1 Like
8

Yes, indeed workflow, crashes, changing apps (we work on 4 different ones), changing for PR-reviews, reseeding, setting changes, etc. I didn’t intend for my example to be taken literally but an illustration of the cost therein.

9

Yes, I saw that, I looked into 1.7.1 times still appeared to be the same, maybe I was missing a magical flag

10

Yes I believe you have to pass a flag into meteor run or it will still do the legacy build.

11

I really wonder how you all manage to get these amazing rebuild times? In all of my Meteor apps, every code change results in about 10-30 seconds of rebuilding (which is pretty frustrating compared to milliseconds in Webpack based apps). Only changes in SCSS files compile a bit faster. Maybe it’s because I’m using a lot of custom packages, which I do because I re-use them across apps? I wouldn’t mind at all about initial build times. Quick rebuild is what I am most looking for.

4 Likes
12

Follow tips here:

Also avoid precompilers (scss, less, sass, coffee, etc), instead if you must transpile them adhoc on your own.

2 Likes
13

When 1.7.1 is ready, you won’t need to use a flag. See the following for the full details (these changes were just merged around the middle of last week, so if you tried the release-1.7.1 branch before then, please give it another shot).

2 Likes
14

@hwillson yeah, I saw that branch, I even tried it on the todos app way before posting this. Didn’t seem to help or do anything, I was running it from a git checkout… I’ll check it again as soon as it’s out. I checked the release-1.7.1.

Still any thoughts on parallelizing the process or avoiding rebuilding on no changes (think switching apps and reusing ports, save mem, easier workflow)

Thanks for the follow up

15

@hwillson @robertlowe I just updated to the latest beta (18 maybe?) and my build times were night and day. They went from like 30+ seconds to 4 seconds. Were there other changes beyond delaying the legacy builds?

16

That’s the big change, but as @benjamn has been working on it, he’s been uncovering several parts of the older codebase that could be improved. He’s making a lot of tweaks to the build system that definitely help speed things up.

17

A few other notable performance improvements:

In short, Meteor 1.7.1 is primarily focused on improving (re)build performance.

The irony is, at first I didn’t want to implement #10055 because it felt like cheating, so instead I found some other ways to (more than) halve rebuild performance first. After that, I realized it was possible to delay the legacy build automatically, without any developer intervention/configuration, which made me more comfortable with that strategy. I don’t love the extra complexity that it adds (e.g. #10077), but cutting rebuild times in half seems worth it.

13 Likes
18

We’re having a relatively slow refresh time, I think fast refresh time is the one of the the most important KPIs for developer satisfaction.

  • We’ve a medium size react app
  • A large number of NPM react components
  • We’ve a package based architecture and each asset add it’s own files (images, css translation)

Our client refresh time is between 14 to 20 seconds, is this normal? Here is the output of the profiler, any insight is appreciated.

| (#4) Profiling: ProjectContext prepareProjectForBuild
| 
| ProjectContext prepareProjectForBuild.........................9,451 ms (1)
| └─ _buildLocalPackages........................................9,299 ms (1)
|    └─ _ensurePackageLoaded(special:cruni).....................9,254 ms (1)
|       └─ IsopackCache Build local isopack.....................9,254 ms (1)
|          ├─ compiler.compile(special:cruni).....................281 ms (1)
|          │  └─ files.withCache..................................265 ms (3)
|          │     └─ compileUnibuild (special:cruni)...............265 ms (3)
|          │        └─ PackageSource#_findSources.................176 ms (3)
|          │           └─ files.withCache                         175 ms (3)
|          └─ Isopack#saveToPath................................8,951 ms (1)
|             ├─ Builder#write....................................239 ms (648)
|             │  └─ files.writeFile                               113 ms (648)
|             ├─ Builder#copyNodeModulesDirectory...............6,306 ms (1)
|             │  ├─ Builder#_ensureDirectory......................546 ms (1557)
|             │  │  ├─ files.stat                                 306 ms (4674)
|             │  │  ├─ files.mkdir                                132 ms (1558)
|             │  │  └─ other Builder#_ensureDirectory             108 ms
|             │  ├─ optimistic readdir                            129 ms (1557)
|             │  ├─ optimistic lstatOrNull........................991 ms (8247)
|             │  │  ├─ optimistic lstat...........................608 ms (8247)
|             │  │  │  ├─ files.lstat                             205 ms (8223)
|             │  │  │  └─ other optimistic lstat                  403 ms
|             │  │  └─ other optimistic lstatOrNull               383 ms
|             │  ├─ optimistic statOrNull.........................265 ms (6690)
|             │  │  ├─ files.stat                                 118 ms (6663)
|             │  │  └─ other optimistic statOrNull                147 ms
|             │  ├─ optimistic readFile.........................2,384 ms (6690)
|             │  │  ├─ files.readFile                           2,164 ms (6690)
|             │  │  └─ other optimistic readFile                  220 ms
|             │  ├─ files.writeFile                             1,567 ms (6690)
|             │  └─ other Builder#copyNodeModulesDirectory        423 ms
|             └─ Builder#complete...............................2,291 ms (1)
|                └─ files.renameDirAlmostAtomically.............2,291 ms (1)
|                   └─ files.rm_recursive                       2,291 ms (1)
| 
| Top leaves:
| files.rm_recursive.......................................2,291 ms (3)
| files.readFile...........................................2,185 ms (7094)
| files.writeFile..........................................1,681 ms (7344)
| files.stat.................................................507 ms (14111)
| other Builder#copyNodeModulesDirectory.....................423 ms (1)
| other optimistic lstat.....................................403 ms (8247)
| other optimistic lstatOrNull...............................383 ms (8247)
| other optimistic readFile..................................220 ms (6690)
| files.lstat................................................206 ms (8250)
| other optimistic statOrNull................................147 ms (6690)
| files.mkdir................................................144 ms (1755)
| files.readdir..............................................110 ms (2162)
| other Builder#_ensureDirectory.............................108 ms (1557)
| 
| (#4) Total: 9,451 ms (ProjectContext prepareProjectForBuild)
| 
| (#5) Profiling: Rebuild App
| 
| Rebuild App...................................................6,593 ms (1)
| └─ files.withCache............................................6,593 ms (1)
|    ├─ compiler.compile(the app).................................523 ms (1)
|    │  └─ files.withCache........................................523 ms (3)
|    │     └─ compileUnibuild (the app)                           523 ms (3)
|    ├─ bundler.bundle..makeClientTarget........................5,444 ms (1)
|    │  └─ Target#make..........................................5,444 ms (1)
|    │     ├─ Target#_runCompilerPlugins..........................472 ms (1)
|    │     │  └─ plugin static-html...............................443 ms (1)
|    │     │     └─ wrapped.fs.readFileSync                       409 ms (625)
|    │     ├─ Target#_emitResources.............................3,877 ms (1)
|    │     │  ├─ PackageSourceBatch.computeJsOutputFilesMap.....1,852 ms (1)
|    │     │  │  ├─ ImportScanner#scanImports for special:cruni.1,159 ms (1)
|    │     │  │  │  ├─ Babel.compile                              365 ms (154)
|    │     │  │  │  └─ ImportScanner#_resolve.....................641 ms (1736)
|    │     │  │  │     ├─ optimistic statOrNull...................440 ms (12581)
|    │     │  │  │     │  ├─ files.stat                           225 ms (2082)
|    │     │  │  │     │  └─ other optimistic statOrNull          170 ms
|    │     │  │  │     └─ other ImportScanner#_resolve            168 ms
|    │     │  │  └─ ImportScanner#scanMissingModules for the app..480 ms (2)
|    │     │  │     ├─ ImportScanner#_realPath                    152 ms (3034)
|    │     │  │     └─ other ImportScanner#scanMissingModules for the app 120 ms
|    │     │  ├─ PackageSourceBatch#getResources................1,397 ms (89)
|    │     │  │  └─ PackageSourceBatch#_linkJS..................1,395 ms (89)
|    │     │  │     └─ linker.fullLink..........................1,332 ms (1)
|    │     │  │        ├─ linker Module#getPrelinkedFiles.........640 ms (1)
|    │     │  │        │  ├─ linker File#getPrelinkedOutput       254 ms (561)
|    │     │  │        │  └─ other linker Module#getPrelinkedFiles 358 ms
|    │     │  │        └─ linker Module#computeAssignedVariables..688 ms (1)
|    │     │  │           └─ linker File#computeAssignedVariables 682 ms (561)
|    │     │  └─ other Target#_emitResources                      558 ms
|    │     └─ Target#minifyJs                                   1,011 ms (1)
|    └─ bundler writeTargetToPath.................................620 ms (1)
|       └─ ClientTarget#write.....................................618 ms (1)
|          ├─ bundler writeFile                                   157 ms (3614)
|          └─ other ClientTarget#write                            362 ms
| 
| Top leaves:
| Target#minifyJs..........................................1,011 ms (1)
| linker File#computeAssignedVariables.......................682 ms (561)
| other Target#_emitResources................................558 ms (1)
| wrapped.fs.readFileSync....................................409 ms (625)
| Babel.compile..............................................371 ms (313)
| other ClientTarget#write...................................362 ms (1)
| other linker Module#getPrelinkedFiles......................358 ms (1)
| linker File#getPrelinkedOutput.............................254 ms (561)
| files.stat.................................................248 ms (2442)
| other optimistic statOrNull................................170 ms (12581)
| other ImportScanner#_resolve...............................168 ms (1736)
| sha512.....................................................132 ms (7010)
| other ImportScanner#scanMissingModules for the app.........120 ms (2)
| 
| (#5) Total: 6,593 ms (Rebuild App)
| 
=> Client modified -- refreshing
=> Finished delayed build of web.browser.legacy in 5334ms
19

Are you by chance not using an SSD or slower SSD, or very large assets? Maybe a lot of pre-compilers?

20

I’m using Macbook Pro, 16 GB ram (could the ram be the issue?), 3.1 Ghz Intel Core i7 with SSD storage. I did remove the assets and it saved I think 3 to 5 seconds.

I think I’ll need to spend sometime to understand how the code base impact the refresh time, but the issue became more noticeable in 1.7.0 due to compiling two bundles, so now we’re on 1.7.1 but it still not optimal.

But is this line normal?
Builder#copyNodeModulesDirectory…6,306 ms

And why is this step needed on refresh!?

Edit: Builder#copyNodeModulesDirectory looks faster with second refresh, 2 seconds…but the average time still greater then 15 seconds unfortunately.

===