[Solved] Meteor 1.8.1 deployed using MUP: JavaScript heap out of memory

Hi everyone,

I know that similar threads exist - but none of them with a solution.

My production server suffers from out-of-memory exceptions which occur multiple times a day.

It is a 32GB machine and I have already set TOOL_NODE_FLAGS to 16GB - but no effect. It always crashes in mongo.js when processing a call from meteor-promise/fiber_pool.js

Does anyone know how to increase the memory limit when using mup? Or is this a known bug? (if yes, is it caused by meteor, mup or node?)

[127.0.0.1]FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
[127.0.0.1]
[127.0.0.1]<--- Last few GCs --->
[127.0.0.1]
[127.0.0.1][1:0x2b974c0] 29149496 ms: Mark-sweep 1315.3 (1461.5) -> 1315.2 (1462.5) MB, 1892.2 / 0.1 ms  allocation failure GC in old space requested
[127.0.0.1][1:0x2b974c0] 29151239 ms: Mark-sweep 1315.2 (1462.5) -> 1315.2 (1431.5) MB, 1743.1 / 0.1 ms  last resort GC in old space requested
[127.0.0.1][1:0x2b974c0] 29153030 ms: Mark-sweep 1315.2 (1431.5) -> 1315.2 (1431.5) MB, 1790.6 / 0.0 ms  last resort GC in old space requested
[127.0.0.1]
[127.0.0.1]
[127.0.0.1]<--- JS stacktrace --->
[127.0.0.1]
[127.0.0.1]==== JS stack trace =========================================
[127.0.0.1]
[127.0.0.1]Security context: 0x26cf01ea58b9 <JSObject>
[127.0.0.1]    1: /* anonymous */ [/built_app/programs/server/packages/mongo.js:~1051] [pc=0x2c7cc48fdcf5](this=0x191599c8c2f1 <JSGlobal Object>)
[127.0.0.1]    2: /* anonymous */ [/built_app/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:~14] [pc=0x2c7cc48fc894](this=0x191599c8c2f1 <JSGlobal Object>)
[127.0.0.1]    3: arguments adaptor frame: 1->0
[127.0.0.1]
[127.0.0.1]==== Details =======================...
[127.0.0.1]
[127.0.0.1] 1: node::Abort() [node]
[127.0.0.1] 2: 0x8ccf9c [node]
[127.0.0.1] 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
[127.0.0.1] 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
[127.0.0.1] 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
[127.0.0.1] 6: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [node]
[127.0.0.1] 7: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::PretenureFlag) [node]
[127.0.0.1] 8: v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int) [node]
[127.0.0.1] 9: v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [node]
[127.0.0.1]10: v8::internal::LookupIterator::ApplyTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>) [node]
[127.0.0.1]11: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [node]
[127.0.0.1]12: v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed) [node]
[127.0.0.1]13: v8::internal::Runtime_SetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [node]
[127.0.0.1]14: 0x2c7cc0c842fd

mup.js:

module.exports = {

  app: {

    env: {
      ...
      TOOL_NODE_FLAGS: '--max_old_space_size=16384',
    },

    docker: {
      image: 'abernix/meteord:node-8.15.1-base',
      buildInstructions: [
        'ENV TOOL_NODE_FLAGS="--max_old_space_size=16384"',
        'RUN export TOOL_NODE_FLAGS="--max_old_space_size=16384"',
      ],
    },
...

There are two different environment variables for

  1. Building the app with Meteor on the computer mup is deploying from
  2. Running the app on the server

TOOL_NODE_FLAGS is used by the Meteor Tool and only applies to #1. You want to modify the --max-old-space-size for #2, which uses the NODE_OPTIONS environment variable instead.

Finally :smile:

Mine is not working and please show me the updated mup file? @sjschmid

There are multiple ways to set the environment variable:

    docker: {
      image: 'abernix/meteord:node-8.15.1-base',
      buildInstructions: [
        'ENV NODE_OPTIONS="--max_old_space_size=16384"',
      ],
      stopAppDuringPrepareBundle: false,
    },
    docker: {
      image: 'abernix/meteord:node-8.15.1-base',
      buildInstructions: [
        'RUN export NODE_OPTIONS="--max_old_space_size=16384"',
      ],
      stopAppDuringPrepareBundle: false,
    },
    env: {
      NODE_OPTIONS: '--max_old_space_size=16384',
    },
2 Likes

I try this way but still FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Hi there, I really have trouble with the deployment:

Minifying app code
<— Last few GCs —>

[64243:0x108008000] 605487 ms: Scavenge 3798.1 (4025.2) → 3783.8 (4025.2) MB, 7.5 / 0.0 ms (average mu = 0.143, current mu = 0.045) allocation failure
[64243:0x108008000] 605544 ms: Scavenge 3836.4 (4062.7) → 3821.1 (4062.7) MB, 4.6 / 0.0 ms (average mu = 0.143, current mu = 0.045) allocation failure
[64243:0x108008000] 605576 ms: Scavenge 3836.4 (4062.7) → 3822.0 (4062.7) MB, 7.1 / 0.0 ms (average mu = 0.143, current mu = 0.045) allocation failure

<— JS stacktrace —>

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Meteor 2.7
Node: v16.14.2

I tried with this
docker: {
image: ‘zodern/meteor:latest’,
buildInstructions: [
// ‘ENV NODE_OPTIONS=“–max_old_space_size=12384”’,
// ‘RUN export NODE_OPTIONS=“–max_old_space_size=12384”’,
‘ENV NODE_OPTIONS=–max-old-space-size=12384’,
‘RUN export NODE_OPTIONS=“–max-old-space-size=12384”’,
],

And also with this in the terminal

export NODE_OPTIONS=“–max-old-space-size=239192”

Over the last months I sometimes got it to work with increasing the number, but now not a chance…

Any ideas?

For anybody else who has this issue, here is how we solved it for the moment.

  1. I ve come to the understanding that the buildInstruction is the wrong place to do this, as this will run in the docker container of your server and not while the app is build locally
    It doesn’t hurt to put in there, but wont solve the problem.

  2. The settings need to be in the Terminal, were you are running the mup file

And these did solve it:
export TOOL_NODE_FLAGS=–max_old_space_size=8192
export NODE_OPTIONS=–max-old-space-size=8192
export JAVA_TOOL_OPTIONS=-Xmx12g
export _JAVA_OPTIONS=-Xmx12g