So over the past month or so, I’ve noticed our Jenkins Galaxy deploy has been getting slower and slower. About a month ago the script was clocking in @ 8 minutes to complete, currently I’m lucky if the job completes before timing out after 20 minutes. Command line deploy still remains at ~8 minutes.
Server specs, disk usage, meteor version, deploy script, project architecture haven’t changed, but the node process is hammering my CPU and hangs at ‘Deploying your app…’.
Gotta be missing something here, any ideas peeps?
Server:
Digital Ocean 4 GB Memory / 60 GB Disk / NYC3 - Ubuntu 16.04.1 x64
Deploy script:
meteor --version
meteor npm install --save crypto request random-color polyline simplesmtp image-downloader gcloud-storage-api babel-runtime phaxio json-2-csv phone-formatter valid-url html5sortable basiccontext digitalocean twilio tabletojson
METEOR_PROFILE=100 meteor deploy myapp.meteorapp.com --settings=$prod_settings
Deploy profile
11:23:49 + meteor --version
11:23:54 Meteor 1.4.2.3
11:23:54 + METEOR_PROFILE=100 meteor deploy myapp.meteorapp.com --settings=****
11:23:58 | (#1) Profiling: ProjectContext prepareProjectForBuild
11:25:02 |
11:25:02 | ProjectContext prepareProjectForBuild........................64,066 ms (1)
11:25:02 | ├─ _initializeCatalog...........................................306 ms (1)
11:25:02 | │ └─ LocalCatalog#initialize...................................301 ms (1)
11:25:02 | │ └─ LocalCatalog#_loadLocalPackages........................258 ms (1)
11:25:02 | │ └─ PackageSource#initFromPackageDir 204 ms (26)
11:25:02 | ├─ _resolveConstraints.......................................53,444 ms (1)
11:25:02 | │ ├─ JsImage#load..............................................270 ms (1)
11:25:02 | │ │ └─ runJavaScript packages/logic-solver.js 167 ms (1)
11:25:02 | │ └─ Select Package Versions................................53,046 ms (1)
11:25:02 | │ ├─ Input#loadOnlyPreviousSolution.........................834 ms (1)
11:25:02 | │ │ ├─ sqlite query 519 ms (176)
11:25:02 | │ │ └─ other Input#loadOnlyPreviousSolution 314 ms
11:25:02 | │ ├─ Input#loadFromCatalog...............................16,244 ms (1)
11:25:02 | │ │ ├─ sqlite query 3,381 ms (242)
11:25:02 | │ │ └─ other Input#loadFromCatalog 12,863 ms
11:25:02 | │ ├─ new CS.Solver........................................1,012 ms (1)
11:25:02 | │ │ └─ Solver#analyze....................................1,009 ms (1)
11:25:02 | │ │ ├─ analyze allowed versions 158 ms (1)
11:25:02 | │ │ ├─ analyze reachability 175 ms (1)
11:25:02 | │ │ └─ analyze constraints 659 ms (1)
11:25:02 | │ └─ Solver#getAnswer....................................34,929 ms (1)
11:25:02 | │ ├─ generate package variables 1,447 ms (1)
11:25:02 | │ ├─ generate dependency requirements 1,958 ms (1)
11:25:02 | │ ├─ generate constraints 26,228 ms (1)
11:25:02 | │ ├─ pre-solve 1,629 ms (1)
11:25:02 | │ ├─ minimize conflicts 1,905 ms (1)
11:25:02 | │ ├─ minimize unanticipated_prereleases 284 ms (1)
11:25:02 | │ ├─ minimize previous_indirect_incompat 267 ms (1)
11:25:02 | │ ├─ minimize previous_indirect_minor 125 ms (1)
11:25:02 | │ ├─ minimize previous_indirect_patch 195 ms (1)
11:25:02 | │ └─ minimize total_packages 282 ms (1)
11:25:02 | ├─ _downloadMissingPackages.....................................419 ms (1)
11:25:02 | │ ├─ Isopack.readMetadataFromDirectory 187 ms (176)
11:25:02 | │ └─ other _downloadMissingPackages 232 ms
11:25:02 | └─ _buildLocalPackages........................................9,862 ms (1)
11:25:02 | ├─ IsopackCache Load local isopack.........................5,152 ms (176)
11:25:02 | │ ├─ files.stat 100 ms (882)
11:25:02 | │ ├─ Isopack#initFromPath.................................4,648 ms (176)
11:25:02 | │ │ ├─ files.readFile 246 ms (528)
11:25:02 | │ │ ├─ files.open 206 ms (1282)
11:25:02 | │ │ ├─ files.read 538 ms (1282)
11:25:02 | │ │ ├─ bundler.readJsImage...............................1,663 ms (9)
11:25:02 | │ │ │ ├─ files.readFile 144 ms (138)
11:25:02 | │ │ │ └─ meteorNpm.rebuildIfNonPortable.................1,442 ms (47)
11:25:02 | │ │ │ └─ meteorNpm.isPortable........................1,378 ms (571)
11:25:02 | │ │ │ ├─ optimistic lstat 357 ms (571)
11:25:02 | │ │ │ ├─ optimistic statOrNull......................361 ms (459)
11:25:02 | │ │ │ │ ├─ files.stat 151 ms (459)
11:25:02 | │ │ │ │ └─ other optimistic statOrNull 209 ms
11:25:02 | │ │ │ ├─ optimistic readJsonOrNull..................409 ms (445)
11:25:02 | │ │ │ │ └─ optimistic readFile.....................322 ms (445)
11:25:02 | │ │ │ │ ├─ files.readFile 213 ms (445)
11:25:02 | │ │ │ │ └─ other optimistic readFile 109 ms
11:25:02 | │ │ │ └─ meteorNpm.isPortable 150 ms (48)
11:25:02 | │ │ ├─ meteorNpm.rebuildIfNonPortable....................1,274 ms (34)
11:25:02 | │ │ │ └─ meteorNpm.isPortable...........................1,236 ms (408)
11:25:02 | │ │ │ ├─ optimistic lstat..............................629 ms (408)
11:25:02 | │ │ │ │ ├─ safeWatcher.watch 275 ms (442)
11:25:02 | │ │ │ │ └─ other optimistic lstat 304 ms
11:25:02 | │ │ │ ├─ optimistic statOrNull 176 ms (337)
11:25:02 | │ │ │ ├─ optimistic readJsonOrNull.....................261 ms (328)
11:25:02 | │ │ │ │ └─ optimistic readFile........................219 ms (328)
11:25:02 | │ │ │ │ └─ files.readFile 152 ms (328)
11:25:02 | │ │ │ └─ meteorNpm.isPortable 108 ms (28)
11:25:02 | │ │ └─ other Isopack#initFromPath 548 ms
11:25:02 | │ └─ other IsopackCache Load local isopack 305 ms
11:25:02 | └─ IsopackCache Build local isopack........................4,621 ms (17)
11:25:02 | ├─ compiler.compile(app-state)..........................3,295 ms (1)
11:25:02 | │ └─ files.withCache...................................3,290 ms (3)
11:25:02 | │ └─ compileUnibuild (app-state)....................3,288 ms (3)
11:25:02 | │ └─ Isopack#ensurePluginsInitialized............3,220 ms (16)
11:25:02 | │ ├─ JsImage#load...............................774 ms (2)
11:25:02 | │ │ └─ runJavaScript packages/ecmascript-runtime.js.681 ms (1)
11:25:02 | │ │ └─ require("./node_modules/meteor/ecmascript-runtime/runtime.js").679 ms (1)
11:25:02 | │ │ └─ require("meteor-ecmascript-runtime") 679 ms (1)
11:25:02 | │ ├─ Npm.require("meteor-babel") 2,113 ms (1)
11:25:02 | │ └─ other Isopack#ensurePluginsInitialized 333 ms
11:25:02 | ├─ Isopack#saveToPath.....................................411 ms (17)
11:25:02 | │ ├─ Builder#write 132 ms (140)
11:25:02 | │ └─ other Isopack#saveToPath 209 ms
11:25:02 | └─ compiler.compile(habitat-support)......................345 ms (1)
11:25:02 | └─ files.withCache.....................................344 ms (3)
11:25:02 | └─ compileUnibuild (habitat-support)................343 ms (3)
11:25:02 | └─ Isopack#ensurePluginsInitialized..............302 ms (24)
11:25:02 | └─ JsImage#load...............................300 ms (1)
11:25:02 | └─ runJavaScript packages/underscore.js 144 ms (1)
11:25:02 |
11:25:02 | Top leaves:
11:25:02 | generate constraints....................................26,228 ms (1)
11:25:02 | other Input#loadFromCatalog.............................12,863 ms (1)
11:25:02 | sqlite query.............................................3,918 ms (430)
11:25:02 | generate dependency requirements.........................1,958 ms (1)
11:25:02 | minimize conflicts.......................................1,905 ms (1)
11:25:02 | pre-solve................................................1,629 ms (1)
11:25:02 | generate package variables...............................1,447 ms (1)
11:25:02 | files.readFile.............................................912 ms (1914)
11:25:02 | analyze constraints........................................659 ms (1)
11:25:02 | other Isopack#initFromPath.................................548 ms (176)
11:25:02 | files.read.................................................538 ms (1282)
11:25:02 | files.stat.................................................432 ms (3083)
11:25:02 | other Isopack#ensurePluginsInitialized.....................334 ms (40)
11:25:02 | other Input#loadOnlyPreviousSolution.......................314 ms (1)
11:25:02 | other IsopackCache Load local isopack......................305 ms (176)
11:25:02 | other optimistic lstat.....................................304 ms (408)
11:25:02 | minimize unanticipated_prereleases.........................284 ms (1)
11:25:02 | minimize total_packages....................................282 ms (1)
11:25:02 | minimize previous_indirect_incompat........................267 ms (1)
11:25:02 | other _downloadMissingPackages.............................232 ms (1)
11:25:02 | other optimistic statOrNull................................209 ms (459)
11:25:02 | other Isopack#saveToPath...................................209 ms (17)
11:25:02 | files.open.................................................206 ms (1282)
11:25:02 | minimize previous_indirect_patch...........................195 ms (1)
11:25:02 | other optimistic readFile..................................176 ms (773)
11:25:02 | analyze reachability.......................................175 ms (1)
11:25:02 | runJavaScript packages/underscore.js.......................173 ms (3)
11:25:02 | runJavaScript packages/logic-solver.js.....................167 ms (1)
11:25:02 | analyze allowed versions...................................158 ms (1)
11:25:02 | other meteorNpm.isPortable.................................152 ms (979)
11:25:02 | other optimistic readJsonOrNull............................129 ms (773)
11:25:02 | minimize previous_indirect_minor...........................125 ms (1)
11:25:02 | files.lstat................................................116 ms (1569)
11:25:02 | files.realpath.............................................101 ms (231)
Usage during deploy:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19843 jenkins 20 0 3711092 2.464g 21308 R 101.3 63.8 3:05.66 node
19053 jenkins 20 0 3564772 638500 20936 S 7.0 15.8 2:02.13 java
1065 root 20 0 276056 3176 476 S 0.3 0.1 0:18.69 accounts-daemon
1301 root 20 0 299772 9848 3284 S 0.3 0.2 2:26.13 docker-containe
1 root 20 0 37944 5300 3288 S 0.0 0.1 0:11.15 systemd