Trying to understand why the compile stage of a rebuild is slow


#1

Hi

I just merged in some code that includes an update from 1.3 to 1.3.2.4 and my rebuild time has gone from 7s to 48s.

Turning the logging on seems to show the issue is in the compile stage, particularly with files.stat and files.lstat. does anyone know how to interpret that into what I need to do to fix this?

| files.readFile 3 ms (3)
| files.exists 5 ms (4)
| Rebuild App…47,749 ms (1)
| ├─ compiler.compile(the app)…42,790 ms (1)
| │ └─ compileUnibuild (the app)…42,790 ms (2)
| │ ├─ files.realpath 2,804 ms (3748)
| │ ├─ files.readdir 4,408 ms (12338)
| │ ├─ files.stat 10,082 ms (69234)
| │ ├─ files.readFile 328 ms (1250)
| │ ├─ files.lstat 18,460 ms (78054)
| │ └─ other compileUnibuild (the app) 6,692 ms
| ├─ bundler.bundle…makeClientTarget…3,722 ms (1)

Things I’ve tried so far, none of which helped:

  • uninstalled and reinstalled Meteor (Windows 8.1)
  • deleted the contents of .meteor/local
  • deleted the contents of node_modules and reinstalled them
  • cleared the npm cache
  • upgrading to 1.3.3 beta
  • deleted everything in AppData\Local.meteor

I also tried reverting back to 1.3 (had to delete everything in AppData\Local.meteor to get that to work) and it’s quicker, it takes around 25s to reload (so still not the 7s or so it was before). As for 1.3.2.4, the delay in 1.3.2 seems to be in the compile stage:

| Rebuild App…24,493 ms (1)
| ├─ compiler.compile(the app)…21,104 ms (1)
| │ └─ compileUnibuild (the app)…21,104 ms (2)
| │ ├─ files.readdir 1,549 ms (5032)
| │ ├─ files.lstat 16,265 ms (78054)
| │ └─ other compileUnibuild (the app) 2,942 ms

I can’t think what else to try?


#2

Have you found a solution yet? I’m having the same issue even on sample 1.3 projects that I clone. I’m getting huge numbers on files.lstat, upwards of 30 seconds.

There is an issue open here, https://github.com/meteor/meteor/issues/4284 that I’m trying to get some action on so feel free to chime in if you find anything.

BTW, I’m compiling on Windows 10/64bit i7, SSD drive and 32GB memory. Can’t believe it’s this slow.


#3

I made some progress; it seems like the number of files in the node_modules directory is at least partly responsible for my issues. More info here: https://github.com/meteor/meteor/issues/7008

My spec’s similar to yours, except Win 8.1 and 16GB memory


#4

Yeah, I’ve been suspecting node_modules for some time. I just did a count and I have 82,000 files in there and that is after doing npm prune


#5

Our app is small at the moment, but I’d anticipate the number of node modules will increase a lot in the future. I think this going to be an issue for a lot of people as 1.3 adoption really takes off. Not sure if it’s just a Windows issue though.


#6

I just tested my same project on my Mac mini (i5/ssd/8gb) and I’m getting 17seconds rebuild time as opposed to 54 seconds on my Win10 machine.


#7

That’s interesting. Maybe add a note about that in https://github.com/meteor/meteor/issues/7008?


#8

Good call Tom, done.