Build Speeds for Development are unreasonable, can we parallelize them or improve them further?

alawi · July 18, 2018, 6:33pm

I just used METEOR_DEBUG_BUILD=1 METEOR_PROFILE=1 meteor run and a phone stopwatch.

And I did some experimentation by moving the the code around to see how it impacts the performance. The suspects were:

Large number of React components/pages
Large node_modules directory
Developing in packages
Custom code we had on package startup
Custom code on server startup

And now I think (1) has added around 6s relative to a fresh meteor project which make sense since we’re about 3 times larger then the starter project and (3) added 7s We’ve some refactoring to do

veered · July 18, 2018, 7:17pm

Try out https://github.com/qualialabs/profile for much much more fine grained rebuild profiles

waldgeist · July 19, 2018, 4:24pm

That’s exactly how my apps are organized. But we don’t use package-level npm modules.

robertlowe · July 19, 2018, 5:00pm

We had a developer do that it was a disaster for various reasons, we instead use imports heavily and organizational level NPM packages, they’re easier to manage and can simply go in package.json. We maintain a few different apps: frontend, backend, and two ternary services. All our apps use vanilla code (js/css/html). We receive burst traffic of 6000+ sessions which maintain for 30+minutes each, we’ve tested our system to 12k+ sessions.

I am constantly on watch for bloat from devs and externally. We/I spend a strong amount of effort keeping meteor lean.

waldgeist · July 19, 2018, 5:19pm

The problem is that npm packages still do not cover all the features I get from Meteor packages. For instance, I have to include different code for web and mobile. With Meteor packages, this can be done pretty easily using the web.browser and web.cordova flags. I would love to see Meteor bringing these must-have features to npm, but unfortunately mobile development doesn’t seem to be the main focus.

BTW: How do you maintain these org level npm packages? Do you host your own npm server, or do you use the npm cloud service?

alawi · July 19, 2018, 5:25pm

Actually the mini-vertical app structures works extremely well since it allows heavy code re-use and shared foundation. So basically instead of splitting the app server/client (horizontally), you do the split vertically. And here I’m only talking about how to structure web apps (platform with mini-apps).

With regards to the refresh/rebuild performance with this architecture implemented via meteor packages, the bottleneck was with the NPM modules in the packages but as @waldgeist hinted it’s better no define them there. Once those NPM definition removed from the packages, the performance of the refresh with and without the package based architecture is almost the same (still high though 8 to 10s due to the large number of page/react components I think).

This image from this article illustrate the concept:

And here is an article detailing the concept, they call it micro-frontends.

xet7 · July 19, 2018, 8:13pm

I use verdaccio for npm package cache. It speeds up meteor builds a lot.

https://www.npmjs.com/package/verdaccio

veered · July 19, 2018, 8:43pm

This helps in development?

xet7 · July 19, 2018, 9:08pm

@veered

Yes. First time when meteor project is build, npm packages are saved locally to harddisk. Next time when building meteor project, verdaccio gets packages from local harddisk, so they don’t need to be downloaded from Internet, so meteor build works faster.

xet7 · July 19, 2018, 9:17pm

I did install verdaccio this way:

npm install -g verdaccio

npm set registry http://localhost:4873/

npm adduser --registry http://localhost:4873

This will create npm registry config file ~/.npmrc

registry=http://localhost:4873/
//localhost:4873/:_authToken="...."

Then start verdaccio in one terminal

verdaccio

And in another terminal build Meteor project

# If you have build script
./build-project.sh
# or alternatively with meteor command
meteor

Then you see on verdaccio terminal how it caches npm packages to harddisk.

If you need to sometime delete npm cache, when verdaccio is not running, deleting is done this way:

rm -rf ~/.npm

xet7 · July 19, 2018, 9:19pm

If you sometime want to not use verdaccio, you can rename .npmrc file, where npm registry address is:

# Change to your home directory
cd
# Rename npm registry file
mv .npmrc .npmrc-not-in-use

robertlowe · July 19, 2018, 10:07pm

Have you considered using dynamic imports? As for NPM we use the NPM service now for easy, but in past we’ve poorly used a “machine-user” as github calls them (a fake/blank user) and OAUTH tokens.

robertlowe · July 19, 2018, 10:13pm

This is a lot of work for something meteor should just handle, and it does, its as you said meteor will pull from disk if it has, but I’ve tested on nvme drives and build times were still the same. The issue is CPU bound, which is thread bound. Webpack has some loader which allow for transpiling in threads, even the Jest testing framework will run in threads. Meteor has to come forward and make use of threading and hide the complexity.

xet7 · July 19, 2018, 11:02pm

I’m still using Meteor 1.6.0.1 still with Wekan, because trying to upgrade newer Meteor https://github.com/wekan/wekan/issues/1522 downgrades coffeescript always when building with meteor command. I did not yet find a way to force specific dependency version, so all this magic in reality distortion field makes me wish for something like Django’s “removing the magic” or Rust’s “language ergonomics”. I am still very grateful about all that Meteor has made easy when maintaining Wekan, and look forward to upgrading for all the new Meteor features. Only way that I can think of to currently speed up builds is using bare metal server that has powerful CPU.

veered · July 19, 2018, 11:20pm

Much of the overhead is in maintaining the in-memory cache. On every reload it looks at every single file to see if it has changed. The actual FS system calls themselves are quite fast, but there is still CPU overhead and memory allocation on the Node side (in many apps, 40% of build times are garbage collection).

Running strace on the Meteor build process is quite interesting. Here is an strace summary for a client-only rebuild in our application:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 75.46    0.304715           0   1388261      6348 futex
  8.44    0.034100           0    234080        59 stat
  5.55    0.022430           0    154309           getdents
  3.53    0.014272           0     77150           openat
  2.34    0.009462           0     94314           rt_sigprocmask
  2.31    0.009318           0     81618           close
  1.42    0.005737           8       709           epoll_wait
  0.50    0.002025           0     26403      4296 lstat
  0.13    0.000541           0      4954       487 open
  0.12    0.000493          27        18           pwrite
  0.07    0.000277           0      4781           read
  0.04    0.000172           0      4449           fstat
  0.03    0.000123           0       345           mmap
  0.01    0.000043           0       122           munmap
  0.01    0.000029           1        40           madvise
  0.00    0.000015          15         1         1 connect
  0.00    0.000015           1        12         5 rmdir
  0.00    0.000015           2        10           unlink
  0.00    0.000012           1        20         1 rename
  0.00    0.000007           1        10           write
  0.00    0.000002           2         1           socket
  0.00    0.000002           0         7         1 epoll_ctl
  0.00    0.000002           1         4         1 inotify_rm_watch
  0.00    0.000001           0         3           recvmsg
  0.00    0.000001           1         2           uname
  0.00    0.000000           0         1           setsockopt
  0.00    0.000000           0         1           getsockopt
  0.00    0.000000           0         7           mkdir
  0.00    0.000000           0         3           readlink
  0.00    0.000000           0         4           inotify_add_watch
------ ----------- ----------- --------- --------- ----------------
100.00    0.403809               2071639     11199 total

Note the massive number of FS system calls. The time spent in the kernel level system call is negligible (which is what strace reports), but the Node-side overhead is large. Looking at an actual CPU profile of rebuilds (using https://github.com/qualialabs/profile) confirms this suspicion.

xet7 · July 19, 2018, 11:43pm

Hopefully new Node v8.12 fill fix some of this, links to performance issues here

veered · July 19, 2018, 11:46pm

Yeah, in particular it’ll fix the crazy fiber switching issues. Will also really help production performance.

Here’s the PR https://github.com/nodejs/node/pull/21593

robertlowe · July 20, 2018, 1:14am

You are better off with an overclocked Desktop CPU as meteor needs single thread performance.

erixtekila · July 22, 2018, 11:16am

Hi Benjamin,
The recent beta version 1.7.1-beta.21 is doing a pretty decent work !
Thanks.

I was quiet deseperate as my build time was getting worse and the application built on meteor 1.7.0.3.
More than 1 minute, each time I was editing a react file in the import forlder.
Meteor server + legacy build, running each time, and taking so long.

Seems that is fixed now, and I can’t thank you more !
Big up.

pgiani · July 25, 2018, 8:33pm

Beta version 1.7.1-beta.21 rocks