New article: background jobs

Hi, very often we find clients struggling with long-running tasks affecting their connected users.

And we have tried a few approaches internally and we decide the share the approach we believe is the best one to solve this problem.

Your feedback is welcome!

10 Likes

Or you could use worker threads/web workers with something like https://threads.js.org/
If only it could be supported by the meteor build tool… :wink:

I use cron.

Hi,

Great to have such documentation on the subject. For my understanding as I’m far to be a specialist in the field, this works if you launch multiple instance of the same app on meteor cloud but this only uses one CPU per instance which is great with meteor cloud (most of them have only one CPU right ?) ?

On the other hand threadJs mentioned above or GitHub - nathanschwarz/meteor-cluster: worker pool for meteor using node js native `cluster` module use the multiple CPUs of the same instance ? (which is better for some other environment)

1 Like

These would really help:

  1. Background Jobs
  2. Cron Tasks
  3. Serverless Functions

Cron is part of the operating system, it’s very easy to use, here’s a user guide below

Bookmark this for writing your crons so you get the timing right

https://crontab.guru

As for severless functions, this is a cash cow for Bezos and cloudt*rds, it still runs on a computer which is a server just you pay for it instead of using the free technologies that are available under gnugpl so you can just setup your own server and get unlimited access, or resell it to CTs that don’t know how to RTFM. It’s a knowledge economy

Cron and background jobs are two different things.

2 Likes

We were using the same approach where crons were running in the user serving container and then decide to migrate all the corns to separate services.
Here’s the blog that I have written about it. Hope this helps.

Interesting article, thanks for sharing.
Been using something quite similar, but we are not satisfied with the basic scheduler provided by aws. What are you using to meet all the scheduler requirements (specially timezone)? Is that a custom service?

We are using meteor schedulers (this is a separate service with no UI) for generating events.

And we are using this package to satisfy our timezone needs.

1 Like

I use cron too for about 5 years without any problem.

For cron jobs, I’m using @vlasky’s excellent percolate:synced-cron. This way I can set up the cron job from inside my Meteor server code.

2 Likes

We’re using job-collections see simonsimcity:job-collection which allows to control how we handle failing jobs, how many jobs of the same type we run and much more. Using it for 6 years now.

This is an old article but it compares packages mentioned here: https://forums.meteor.com/t/jobs-worker-in-meteor-steve-jobs-vs-meteor-jobs-vs-meteor-workers-vs-synced-cron/44578

we used the exact same approach in some meteor apps.

this has the benefit that you can use existing code of your app.

system cronjobs are a bit trickier in pure meteor apps, as its harder to share code between the cron-script and the webapp.

in general, avoid running cpu intense tasks on the same process that does answer web-requests and instead run as separate worker-process

2 Likes