Meteor takes more than 15 minutes to build / rebuild - what am I doing wrong?

Hi,

I’m the CTO of a new, stealth-mode company in the retail space.

We were about to go ahead with ReactionCommerce as our framework (which is heavily dependent on Meteor)

As we started implementing a P.O.C, we noticed a very very slow build and rebuild workflow, that often will take more than 15 minutes, even for the slightest code change.

I ran meteor with a profiler and encountered the following bottle-necks -

1. Babel Compile :

Babel.compile…306,181 ms

2. prepareProjectForBuild Method :

ProjectContext prepareProjectForBuild…208,197 ms (1)
| ├─ _downloadMissingPackages…756 ms (1)
| │ ├─ Isopack.readMetadataFromDirectory…337 ms (133)
| │ │ └─ files.readFile 280 ms (133)
| │ └─ other _downloadMissingPackages 419 ms
| └─ _buildLocalPackages 207,337 ms (1)

Any idea on how we can resolved that? it’s pretty much a make-or-break for us…

Thanks!

We have a very large (in terms of number of files and dependencies) Meteor app.

Meteor has become very fast in recent years. We have never been unhappy with the build times, or the restarting delay, … until recent BIOS updates on dev machines relating to Meltdown and Spectre, that is. Because Meteor in development is IO intensive, you are very likely to notice a change in speed.

I’m thinking that that’s what’s biting you. YMMV depending on the CPU generation.

@illustreets is that really a possible issue? the BIOS updates on dev machines? it seems too down the stack for me to cause an issue.

I’ve been seeing a lot of complaints about meteor’s buid and rebuild speeds - you mentioned you were never unhappy - how long does meteor takes to build for you?

Yes, it is the most probable possibility. One after another, each machine got slowed down to a crawl after the latest BIOS update - we use Linux on Thinkpads. Strong machines, originally. Not only Meteor, but the entire host of other services we use along with the Meteor app (e.g. a multi-threaded Java service for tiling vector geometries).

Before this, I don’t know exactly, maybe a minute at worst (on Core i7, SSD, 32GB RAM). Nowadays I have a long enough break for making coffee, and even prepare a little nosh. Like you, around the 10 min mark.

The ‘Meteor is slow’ stories are outdated. And maybe also people not having decent machines.

A very clear example of how big the impact of these patches is on IO heavy systems (3x CPU utilisation!!): https://www.epicgames.com/fortnite/forums/news/announcements/132642-epic-services-stability-update

As explained in an Ars:

A game server will typically run as on a dedicated machine, as the sole running process, but it will perform lots of network I/O. This means that it’s going from “hardly ever has to flush the TLB” to “having to flush the TLB thousands of times a second.”

Ouch - these build times are not normal, even for a large application. Something is definitely wrong. Please open a GH issue for this describing your application setup and include profiling results (METEOR_PROFILE=1 meteor). It would be even better if anyone experiencing slow build times can share their full application for troubleshooting (but I completely understand how that’s not always possible).

2 Likes

I understand how that can be worrying. In our case we didn’t even bother signaling the issue, since everything we use got slower with each BIOS update, on each machine. Meteor is only one of the applications that are affected.

We are working with the hardware manufacturer to solve the issue.

It’ll be interesting to see if there’s a solution, even if only for Meteor.

Which OS do you have these build times on?

Anti-virus (which mostly means Windows machines) scanners slow down Meteor a lot, like 300x slower, so it could be that

@coagmano I’m on Mac OS Sierra.

I disabled my AVG antivirus file scanning but the issue persists.

Can you try running Meteor as:

TOOL_NODE_FLAGS="--max-old-space-size=8192" meteor

to see if that makes a difference?
(assuming of course you have that much memory available, if needed.)

I’m seeing similar performance, where the buildLocalPackages step takes most of the time.
I took CPU profiles and it turns out all that time is spend by the garbage collector. So it seems to be a memory problem. What @hwillson proposes might help, but it depends if your hardware can handle it. It’s still not a solution, but a possible workaround.

For more info, see: https://github.com/meteor/meteor/issues/9568

Also try what benjamin asks here: https://github.com/meteor/meteor/issues/9568#issuecomment-382537792 . I don’t have a conclusive benchmark with that change yet, but the more data, the bigger the chance this can be solved.

2 Likes