Minification running out of memory after adding npm package

help
1

I recently added the NPM package @netless/fastboard

I can run locally or deploy with --debug flag no problem.

However running meteor --production or a meteor build or meteor deploy (without --debug) the app almost always gets stuck on the minification process. It just keeps minifying forever, until JS is out of memory. If i give it more memory to work with, it takes longer to crash but ultimately crashes with the same out of memory message.

I say “almost always” because in maybe ~3% of cases it takes long but eventually succeeds, the other 97% it fails

How can I debug this further, or solve this?

2
1 Like
3

Thanks a lot @paulishca !

I’ve tried it and it doesn’t print anything for the minifier when it crashes unfortunately. First it shows everything is fine:

| (#4) Profiling: ProjectContext prepareProjectForBuild
| 
| ProjectContext prepareProjectForBuild                           126 ms (1)
| 
| Top leaves:
| 
| (#4) Total: 126 ms (ProjectContext prepareProjectForBuild)
| 
| (#5) Profiling: Rebuild App

Then it just shows the same error it already showed before, which is

<--- Last few GCs --->

[25270:0x102d7e000]  1312259 ms: Mark-sweep 3926.6 (4127.7) -> 3922.4 (4129.9) MB, 6119.3 / 1.3 ms  (average mu = 0.103, current mu = 0.008) allocation failure scavenge might not succeed
[25270:0x102d7e000]  3114473 ms: Mark-sweep 3932.7 (4129.9) -> 3928.6 (4132.9) MB, 1802175.5 / 0.2 ms  (average mu = 0.000, current mu = 0.000) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x100a0ddd9]
    1: StubFrame [pc: 0x100994b41]
Security context: 0x2f133d5408d1 <JSObject>
    2: pushContext [0x2f1324f347a1] [/Users/bart/meteor/packages/standard-minifier-js/.2.6.0.1h0zsqi.ostbg++os+web.browser+web.browser.legacy+web.cordova/plugin.minifyStdJS.os/npm/node_modules/meteor/babel-compiler/node_modules/@babel/traverse/lib/path/context.js:~211] [pc=0x1855dd9de64d](this=0x2f14505d2681 <NodePath ma...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

That said, I just tried to create a new minimal blaze app and the errors are different. With METEOR_PROFILE=100, it first says that other Target#minifyJs takes 163,543 ms:

|  Starting your app                         /
| files.stat                                                        0 ms (5)
| files.unlink                                                      0 ms (1)
| files.readFile                                                    0 ms (5)
| files.writeFile                                                   6 ms (1)
| Build App...................................................185,314 ms (1)
| └─ bundler.bundle...........................................185,313 ms (1)
|    ├─ compiler.compile(the app)...............................1,940 ms (1)
|    │  └─ compileUnibuild (the app)............................1,939 ms (4)
|    │     └─ Isopack#ensurePluginsInitialized..................1,636 ms (56)
|    │        └─ JsImage#load                                   1,632 ms (6)
|    ├─ bundler.bundle..makeClientTarget......................180,971 ms (3)
|    │  └─ Target#make........................................180,971 ms (3)
|    │     ├─ Target#_emitResources............................14,369 ms (3)
|    │     │  └─ PackageSourceBatch.computeJsOutputFilesMap....14,115 ms (3)
|    │     │     ├─ ImportScanner#scanImports for ecmascript-runtime-client 1,065 ms (3)
|    │     │     ├─ ImportScanner#scanImports for base64        1,200 ms (3)
|    │     │     └─ ImportScanner#scanImports for the app.......9,479 ms (3)
|    │     │        └─ ImportScanner#findImportedModuleIdentifiers 9,134 ms (98)
|    │     ├─ Target#minifyJs.................................164,975 ms (3)
|    │     │  ├─ Npm.require("terser")                          1,189 ms (1)
|    │     │  └─ other Target#minifyJs                        163,543 ms
|    │     └─ ClientTarget#minifyCss............................1,471 ms (3)
|    │        └─ minifyCssFiles.................................1,471 ms (3)
|    │           └─ CssTools.minifyCss                          1,277 ms (3)
|    └─ bundler writeSiteArchive................................1,410 ms (1)
|       └─ bundler writeTargetToPath............................1,265 ms (4)
|          └─ ServerTarget#write................................1,167 ms (1)
|             └─ JsImage#write                                  1,141 ms (1)
| 
| Top leaves:
| other Target#minifyJs..................................163,543 ms (3)
| CssTools.minifyCss.......................................1,277 ms (3)
| files.stat...............................................1,168 ms (7965)
| files.watchFile............................................507 ms (11460)
| files.readFile.............................................493 ms (2719)
| Babel.compile..............................................334 ms (292)
| other minifyCssFiles.......................................185 ms (3)
| ImportScanner#findImportedModuleIdentifiers................144 ms (60)
| other bundler writeSiteArchive.............................138 ms (1)
| other PackageSourceBatch.computeJsOutputFilesMap...........113 ms (3)
| files.writeFile............................................112 ms (288)
| sha1.......................................................108 ms (3324)
| 
| (#3) Total: 185,320 ms (Build App)
|

And then crashes with a different error than my main app:

=> Errors prevented startup:                  
   
   While minifying app code:

   /Users/bart/.meteor/packages/standard-minifier-js/.2.6.1.1i7bkdp.ef1s++os+web.browser+web.browser.legacy+web.cordova/plugin.minifyStdJS.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-plugin-minify-mangle-names/lib/scope-tracker.js:47:34:
   unknown: Cannot read property 'add' of undefined while minifying packages/modules.js
   at ScopeTracker.addReference
   (/Users/bart/.meteor/packages/standard-minifier-js/.2.6.1.1i7bkdp.ef1s++os+web.browser+web.browser.legacy+web.cordova/plugin.minifyStdJS.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-plugin-minify-mangle-names/lib/scope-tracker.js:47:34)
   at ReferencedIdentifier

.....  many many similiar lines ...

   (/Users/bart/.meteor/packages/standard-minifier-js/.2.6.1.1i7bkdp.ef1s++os+web.browser+web.browser.legacy+web.cordova/plugin.minifyStdJS.os/npm/node_modules/meteor/babel-compiler/node_modules/@meteorjs/babel/index.js:185:33)
   at Object.minify (packages/babel-compiler.js:59:29)
   at meteorJsMinify (packages/minifier-js.js:60:25)
   at packages/minifyStdJS/plugin/minify-js.js:126:20
   at Array.forEach (<anonymous>)
   at MeteorBabelMinifier.processFilesForBundle (packages/minifyStdJS/plugin/minify-js.js:118:9)
   
   
=> Your application has errors. Waiting for file change.
4

Potential problems:

  1. memory leak on the minifier
  2. memory leak on your app (although I am not 100% sure if this should manifest during build)
  3. memory leak with babel/plugins or in other compilers like typescript
  4. more memory required by node (unless the device lacks memory, increasing memory allocation sometimes only hides the memory leak issue)
1 Like
5

Could you please look into this and give it a try: Using npm Packages | Meteor Guide for your @netless/fastboard package. I had a look and I see it depends on 2 packages: core and UI. The UI depends on tippy.js which I remember I god rid of but I cannot remember why but most likely because of compiling issues :).
If compiling with Meteor doesn’t work, you may try to get this package locally and modify it to depend on V5 of tippy.js. I think Tippy might not be essential to your use. It is just a programable popper but a popper after all.

1 Like
6

Great ideas, thanks a lot!

I’ve tried a few variations of this but unfortunately it didn’t solve the issue in this case.

I am currently updating the app’s Meteor version as it was stuck on 2.2, I have some signs that may fix the problem, I will report back.

7

Hi Bart, did you try to increase the heap size?

There are a few Meteor apps that I’ve worked with that they need 6GBs of Node.js max heap to be able to build.

In your case you are getting OOM with 4GBs which is the standard for Node.js 14

Let me know if you need help with this.

2 Likes
8

Another option is trying the beta version of zodern:standard-minifier-js. It uses swc to minify, which uses less memory, and avoids the error you are sometimes seeing with the ScopeTracker.

1 Like
9

@filipenevola thank you! I did try to increase it to a much bigger value and it didn’t solve the issue unfortunately

@zodern thank you very much, I’m going to try it and will let you know how it goes!

10

I’ve tested this beta with a client project that was not working with the previous version and it worked fine :slight_smile:

1 Like