Not sure how to go about this…
I have a background task that needs to run every 12 hours. It takes 30 minutes to an hour to complete. What it does is it has a loop that will call an API for an x amount of times (I think right now it has about 500 calls, will differ depending on the API response), retrieve the response, process it and either overwrite existing or insert new document into mongo.
Currently I have a cron set up with percolate:synced-cron currently which works fine on my local machine build, however when I upload it to Galaxy hosting, the same cron job stops with this error:
Application exited with signal: killed (out of memory)
So firstly, how can I avoid the out of memory issue?
And secondly, would it be better to run a separate application which only purpose is to update the data in mongo? Im not sure how to do this either as this would just be a background task with no frontend… Just a simple nodejs server without meteor?
Any help appreciated