[solved] Manage cron workload


#1

We are using https://www.npmjs.com/package/cron and the crons are working, but some times the workload of the cron creates issues on the site due to the workload.

What is the best way for a cron to impact as little as possible the site?
Could it be possible to have one container a container in Galaxy dedicated to run the crons?


#2

What exactly is the cron doing?

I have a couple of crons running on a baby production project (every 30 seconds), with basically zero impact on site performance (I’m using a single galaxy container, their smallest one).

That said, I have in my mind a mental to-do to make this more efficient in the future (when must-needs…), by way of possibly using lambda or a similar FAAS. But my cron function is really simple…


#3

Oh also, this looks promising by: https://github.com/msavin/SteveJobs..meteor.jobs.scheduler.queue.background.tasks

I haven’t got round to reconfiguring my code base to use it yet, but on the to do list as well!


#4

The cron that is making some issues is the one checking the emails. Once it find a new email it has to do some process with it.

Thank you for the feedback, we will check the packages :slight_smile:

Cheers


#5

How often is it running the cron?

And what is it doing re: checking the emails?


#6

Is running every minute,

  • check for new emails
  • copy the content and attachments
  • removes the email

#7

Networking speeds in hosted environments are extremely slow. You’re probably losing a lot of time downloading an attachment.


#8

not sure about galaxy,

but we have a setup on our own kubernetes cluster where one pod / container is dedicated to run the crons.

I simply added an additional environment variable there and check on startup, if this env var is set. If so, or the app happens to run under develop locally, i start the cronjobs.


#9

We use AWS Lambdas for CRON jobs, including webhooks for services such as emails (where we receive, treat and store emails).


#10

Is the Meteor instance responsible for running the cron job the same as your webapp instance ?

If it is the case, you can add in your infrastructure a second Meteor instance dedicated to cron tasks sharing the same code and same access to your DB.

You can also move the workload of sending emails to another node/Meteor instance using messages queues (RabbitMQ/amqp.node/) (with RabbitMQ & the second meteor instance on a another server)


#11

Yes, the issue was the connection to an specific email server.

Thank you for all the different options on how to work around the issue.

Cheers,