[solved] Manage cron workload

We are using https://www.npmjs.com/package/cron and the crons are working, but some times the workload of the cron creates issues on the site due to the workload.

What is the best way for a cron to impact as little as possible the site?
Could it be possible to have one container a container in Galaxy dedicated to run the crons?

What exactly is the cron doing?

I have a couple of crons running on a baby production project (every 30 seconds), with basically zero impact on site performance (I’m using a single galaxy container, their smallest one).

That said, I have in my mind a mental to-do to make this more efficient in the future (when must-needs…), by way of possibly using lambda or a similar FAAS. But my cron function is really simple…

Oh also, this looks promising by: https://github.com/msavin/SteveJobs..meteor.jobs.scheduler.queue.background.tasks

I haven’t got round to reconfiguring my code base to use it yet, but on the to do list as well!

The cron that is making some issues is the one checking the emails. Once it find a new email it has to do some process with it.

Thank you for the feedback, we will check the packages :slight_smile:

Cheers

How often is it running the cron?

And what is it doing re: checking the emails?

Is running every minute,

  • check for new emails
  • copy the content and attachments
  • removes the email

Networking speeds in hosted environments are extremely slow. You’re probably losing a lot of time downloading an attachment.

not sure about galaxy,

but we have a setup on our own kubernetes cluster where one pod / container is dedicated to run the crons.

I simply added an additional environment variable there and check on startup, if this env var is set. If so, or the app happens to run under develop locally, i start the cronjobs.

We use AWS Lambdas for CRON jobs, including webhooks for services such as emails (where we receive, treat and store emails).

Is the Meteor instance responsible for running the cron job the same as your webapp instance ?

If it is the case, you can add in your infrastructure a second Meteor instance dedicated to cron tasks sharing the same code and same access to your DB.

You can also move the workload of sending emails to another node/Meteor instance using messages queues (RabbitMQ/amqp.node/) (with RabbitMQ & the second meteor instance on a another server)

Yes, the issue was the connection to an specific email server.

Thank you for all the different options on how to work around the issue.

Cheers,