Fibers: RangeError: Out of Memory

help
1

I’m using fibers to make thousands of HTTP calls. When it gets to about a 1000 or more calls, Meteor gives the Out of Memory Error.

Here’s is the code for my function:

    var urlCounter = -1;
    function getRequest() {
      urlCounter++;
      var res = HTTP.get(theURL[urlCounter]);

      return res.data;
    }

    var futures = _.times(theURL.length, function() {
      // Wrap the call in a future to execute in a new fiber.
      return Future.task(getRequest);
    });
    // Wait for all the futures to complete.
    Future.wait(futures);
    // Gather the results.
    var theResults = _.map(futures, function (f) {
      return f.get()
    });

Let’s assume theURL holds 1000 different URLs, so I’m going through the array using urlCounter.
theResults holds the returned results from the HTTP calls.

When the function is run too many times, for example >1000, then I get the following error:

C:\Users\Computer\application\node_modules\fibers\future.js:471
                            }).run();
                              ^
RangeError: Out of Memory 
at RangeError (native) 
at C:\Users\Computer\application\node_modules\fibers\future.
at nextTickCallBackWith0Args (node.js: 489:9)
at process._tickCallBack

How can I fix the fibers so I can make the thousands of calls in the least amount of time without getting the out of memory issue?

1 Like
2

Not sure I can answer that, but I can make the following observations:

  • Any call, especially an asynchronous call` has a large memory overhead for the maintenance of the closure. For an asynchronous function, that memory cannot be released until the asynchronous call is completed. Furthermore, the memory will not be released until the garbage collector runs.
  • var res = HTTP.get(theURL[urlCounter]); - that’s already running in a fiber, so it looks like you’re running fibers within fibers.
  • The use of urlCounter external to the function is a bit of an anti-pattern.

You may be able to “batch” your calls into chunks which do not cause memory to be exceeded, although that will lengthen the end-to-end time. You could also look at using async and await - but they are also memory hungry and I’ve never looked at their memory efficiency when compared with fibers.

It may also be worth looking at tuning Node.js's garbage collector, but you may find it easier to just bump the memory up (although that will likely just delay the inevitable).

1 Like
3

I’ve batched the calls into chunks of 500 and that seems to have helped a lot but is not giving me quite the performance I am looking for.

When you say that var res = HTTP.get(theURL[urlCounter]); is already running in a fiber, does that mean that I can run these HTTP.get requests in a for loop and get the same results performance wise?

4

No. Running in a fiber means that they run sync-style - each has to complete before the next is started. If you want to make an http request without waiting you need to use the callback form. This gist uses the old form of Meteor’s http package (Meteor.http rather than HTTP), but you may find the content helpful - you’ll notice the use of the callback.

Alternatively, you may find that wrapping the callback form in a Promise, setting up an array of Promise requests and using await Promise.all(arrayOfPromiseRequests) just works!

5

I just tried using the code with the callback but it doesn’t seem to work too well when there are parameters in the HTTP request.

var futures = _.map(urls, function(url) {
    var future = new Future();
    var onComplete = future.resolver();

    /// Make async http call
    HTTP.get(url, 
    {
      headers: {
       'Content-Type': 'application/json'
      }
    }, function(error, result) {

      // Get the title if there was no error
      var title = (!error) && getTitle(result);
      onComplete(error, title);
    });

    return future;
  });

  // wait for all futures to finish
  Future.wait(futures);

  // and grab the results out.
  return _.invoke(futures, 'get');
6

There should be no difference with parameters - the non-callback (fiber) form of the get uses the callback form under the covers.