So I have the following scenario -
I have a set of Mongo records (10,000+) returned by a Meteor cursor. I need to iterate through each of the records, pull a bit of data, send that bit over to an external API, then update the record based upon the response. Each API call takes about 2 seconds on average.
I have my program working to process individual records, but the forEach loop of the cursor executes and attempts to execute all 10,000 requests at once. Of course, I don’t have enough memory or bandwidth to execute 10,000 simultaneous API requests, so I run out of resources. This is also not what I want to do, anyways.
Time is not really a concern - so long as the program is running - this is not a user facing process.
What I am trying to achieve is one of the following
a) Setup the system so that it only calls the API when the last request is received or timed out. There will never be more than one request open at once
b) Setup the system so that I can have n threads working on the list [using the last digit in a document number to determine thread]
So I have been looking at fibers and futures and trying to wrap my head on them - any help or suggestions would be appreciated!