Best way to throttle a bunch of HTTP API requests? (or is this even needed)

General question here because I am sure other people have sorted this one out.

I am running a synced-cron request once per day and looping through my DB and this can trigger a bunch of HTTP API calls. I want to avoid locking up my app server doing this all in one bang - because there could be up to 1,000 (unlikley) of these requests. My HTTP calls are blocking on the server (ie. I don’t use a callback for the result).

A simple solution could be that I make the cron jobs function build 20 or 30 different queues and execute those with interval controlled setTimeouts… eg. loop over the database, build the queues and call each of them with setTimeouts in 5 or 10 minute increments.

The other option, but I don’t know if this is a good one would be to just do the loop over all the requests and for each individual request do a setTimeout with +1 minute over the previous request. As I said, there is no risk of this queue being 100,000 records… max is 1,000. however I don’t know how well the server will cope with say 1,000 setTimeouts running at the same time… is that going to be worse than just hitting the loop without bothering for this sort of micromanagement of tasks?

I’ve been running some tests and because my calls are blocking it does seem to slow the process down on its own and it doesn’t lock up the CPU on the server either. So I guess I answered my own question, I don’t need to worry.

Queueing is absolutely essential for safer, decoupled and scaled inter-server communication. 1,000 timeouts should not be a problem at all, I have run simulations that use 100k timeouts with no problem. I wrote this library a few months ago: for the purpose of hitting arbitrary APIs in a way so as to not invoke rate-limit restrictions.

It’s not really production-ready and I am not working on it right now, but it is “done” and it “works”. The main idea is for a bunch of distributed workers to be able to atomically consume a job+key pair at any time. A job is a function name and arguments, a key is an api-key-type object which contains a rate-limit and an id. Workers are capped queues (you set N) and will consume N jobs and the keys, and then queue up the jobs and execute them when a key becomes available. A key becomes available when the timeout disappears. A key with a rate limit of 10 calls per second has a timeout of 100ms, so the worker will execute a job, mark the key as cooling-down, and then a timeout after 100ms will mark the key as active again, and the worker consumes the next job as soon as a key becomes active.

You might want to implement a queue, or use an existing queue library or job-collection, or whatever. You could consider doing batched jobs on AWS, which seems a lot easier.