I’ve been using Meter Jobs now for quite a while and know some of it’s limitations. I’ve recently seen the raise of the package Steve Jobs and found Meteor Workers and Synced Cron on my way now.
@msavin what was your main motivation for developing Steve Jobs? Did you know of the other packages? If yes, where did you see their shortcomings?
Who knows the other packages and can push in to a comparison where users can find the best package for their need?
I’ve chosen Meteor Jobs quite some time ago and chose to stick around with it. Recently, since it’s author announced to put it into maintenance and Meteor 1.6 required some changes, I chose to open my own fork and maintain it because the original author didn’t merge it in until now.
There’s still much to dive into and learn, but I guess the audience of developers has opened up since I threw out coffeescript
If nobody here is able to convince me of any better system, I’ll continue modernizing it and adding tests to ensure it’s functionality in future releases.
We use meteor jobs in one of our projects. It’s possibly a bit complex for some users but it’s far more feature rich than SyncedCron (and I’ve not tried SteveJobs). It certainly has the odd known bug in it and the fact that it’s coffee makes it’s that bit more awkward but it does a good job. I didn’t know you were maintaining a fork @simonsimcity . What are your main updates ? Have you created this poll to see if it’s worth supporting it further ? We’ll look at switching to your fork.
I just checked the version we’re at actually and it’s stuck at 1.4.0 even though there’s a 1.5.2.
More than anything, I wanted something that fits the ergonomics of Meteor. You should be able to drop in a couple of snippets, fill them with your code, and be good to go.
I took a look at what’s out there, and it seemed either:
outdated / unmaintained / had lots of issues
overly complex API
strange (i.e. exposing a bunch of stuff on the client instead of just giving developers the option to do it themselves).
As for configuration/capability, I’d say Steve Jobs is the as capable, if not more capable, than all the other queues.
As for performance/reliability/etc, for many reasons I think MongoDB is a better choice than Redis. I even read that MongoDB outperforms Redis when the data volume goes beyond a certain point. Plus, you can use a memory-based storage engine for MongoDB if you really need performance.
As for limitations. I think the biggest right now is that:
1- all the jobs are stored in one collection, but they should be stored in individual collections. Good news is, its an easy change to implement.
2- the jobs run consecutively, and one at a time. However, this too could be upgraded, and it can also be spun as a feature.
As for opportunities, I think the biggest right now are:
1- supporting transactions for cases where you need to spin up new jobs out of one job
2- building some kind of in-memory cache of jobs to multiple jobs as once, and perhaps integrate Change Streams to alert the queue if a job had been terminated on the last second
Both of these items largely depend on Meteor upgrading the MongoDB driver to support these new features.
The author announced that the project would go into maintenance mode from the beginning of 2018.
A call for updating the coffeescript dependency (which was needed to stay compatible with the upcoming Meteor 1.6.1 release) made it necessary to change some parts of the code (see: https://github.com/vsivsi/meteor-job-collection/pull/268). @mitar was very active in this process and also I tried to push it for quite some time until I decided to fork off. My version currently is available at https://atmospherejs.com/simonsimcity/job-collection. The version 1.6.1 only includes the upgrade to coffeescript 2.0, whereby I later on spent some time to fully get rid of coffeescript after all. I haven’t published this version yet because I don’t see a reason to push it out sorely for dropping the dependency.
This poll was a try to listen to the community if it’s worth supporting the Meteor Jobs package in the future or if I should abandon it and move on. Maybe there’s something better and now is the time to switch before putting a lot of effort in something that will be left aside anyways.
Since you’re using a plain node package, do you see any downside? Are you using it in a Meteor project or as a decoupled vanilla NodeJS application? When using it in Meteor, are fibers getting you into any trouble here?
I am using it in a Meteor project without issues. What I love about beequeue is that it is a small package and not increase the size of my client package (comparing to Meteor Jobs). This was the biggest reason why I moved (from 1/3 my client code build to negligible)
Cons:
not suitable for long running jobs
no repeatable jobs
I’m currently looking at another jobs package that can add these 2 things only in the server.
So @simonsimcity, dropping Coffee would be nice. We’d like to eliminate it entirely from our codebase but there are still a handful of packages that use it. Have you been able to test the pure JS version at all? It’d be great if you were supporting meteor-jobs as we’re quite heavily invested in it.
Looking at the alternatives on npm I think I would opt for Bull if I had to start from scratch. Bee-queue seems nice but limited.
I’ve had actively evaluated bee, bull and kue about 6 months ago, and decided to go with bull. It is quite feature rich, well maintained and easy to set up and integrate into your application. Of course, one important downside might be the hard dependency on a redis instance, which you might find undesirable for relatively smaller applications.
Although if you are on amazon or google cloud, I’d prefer their own pub/sub service offerings to build a job queue infrastructure. They do lack the job-centric feature sets, but feel more reliable due to their real time, scalable, no-devops nature.
I’ve used meteor:workers successfully for several years, and it’s been fairly reliable. Since it depends on SyncedCron, I’ve got that too, though it seems to have some trouble with timezones. My goals were always to have a simple, Meteoric solution that didn’t require standing up extra databases or coordinating multiple Git repositories, so these packages have been great for just forking ffmpeg, etc. processes in the background without freezing up the web app.
The main issue I have now with meteor:workers is its CoffeeScript dependency. I’ve been leaning towards TypeScript lately, and there’s just enough mismatch between the two to be irritating.
Slightly OT: What I’d always been hoping for, which I haven’t seen yet, was a jobs system with a bit more ability to coordinate ‘meta-jobs’. Something like Apache Airflow, but written as a Meteor package. I realize this bumps up the complexity quite a bit compared to a simple queue, but just wanted to put in my $0.02 in case any beneficent Meteor devs have too much time on their hands and are looking for something fun to work on
I have some experiences related to this, but I might have overthinked this a lot. Short answer is that we had to use the NPM package Agenda.js to make Jobs/Workers function well.
We have an Meteor app, which frequently must sync data with a external API. We do this each 15 minutes. Since the external API does not support Webhooks etc, we must sync each 15 minutes. This is a lot of data, since it must sync for each “project” we have in the app. So 100 projects will result in 100 API calls and syncs. The API only supports XML, so we have to parse XML->JSON via a NPM package too. 100 times.
All of this runs each 15 minutes. As for now, we are just using Synced Cron package to run these jobs each 15 minutes. It will each 15 minutes call a function which syncs all the “projects” one at a time. It works fine, but lately its being slower and slower. It also blocks the event loop, due to poor code, so the app are being a bit slow when this runs.
We decided to rewrite this, and I’ve tried many packages before finally landing on NPM module Agenda.js. Since we are using Heroku for hosting, we want to utilize the worker dyno functionality for this. Heroku will then spin up own worker dynos which are separated from the web dynos, which will scale well and not harm the performance on the web dynos. I’ve written a pure Node file, which does the actual sync process together with Agenda. This pure Node file are added to the worker dynos via Herokus Procfile file.
To make this work with buildpacks etc, we have to modify the buildpack to requre this file in the Procfile.
Then all of this should work very well for us. We haven’t finished it yet, but are in the final steps of the process.
But we aim for performance since we have heavy running jobs, so apps with much lesser heavy jobs should be fine with e.g. Steve Jobs package.
Happy user of Steve Jobs, does the job very efficiently for the project where I needed it and @msavin was always very helpful and responsive on the forum which for me is a big plus (actually the biggest plus).
Steve Jobs won’t handle as much volume as fast as other queues - at least not until I or someone upgrades its operator features - but it can do heavy jobs as well as any queue, if not better.
Take a look at this “staged” job. The idea is, this job has has multiple steps, and we need to verify that each one of them has completed. If a server crashes in the middle of a job, the next server could pick up where the previous one left off.
Jobs.register({
"stagedJob": function () {
var self = this;
var setStage = function (stage) {
var state = self.set("stage", stage)
runStage()
}
var runStage = function () {
var stage = self.get("stage")
switch (stage) {
case 1:
// add your logic here
setStage(2)
break;
case 2:
// add your logic here
setStage(3)
break;
case 3:
this.success()
break;
default:
setStage(1);
}
}
runStage();
}
})
Here, we are using this.set to store the state for each job with-in the database, and this.get to get that value. In your case, you can use that to keep track of what, and how much of, data has synced, and ensure that it is executed thoroughly.
But Steve Jobs queue is not worker based, I assume? When you have long running jobs, which might also block event loop, this will also block the same event loop the web process are running in, which will make the app laggy for users when running jobs.
This is why I have been looking for a bit custom solution with Agenda.js and workers in Heroku. (workers are just other dynos which run other processes than your main web process)
I can now bring some results after my testing:
I have modified the Meteor Buildpack Horse to support workers in Heroku. Here is my fork: https://github.com/invisonybygg/meteor-buildpack-horse This actually just modifies the Procfile to run a defined file as a worker, e.g. worker.js. Since Meteor bundles everything into a main.js file, you have to place the worker file inside a /private folder, which are interpreted as assets. This are not bundled after a build, and works perfectly to place worker files.
Then you can scale up workers in Heroku after your desire.
Since Heroku is a bit weird, it needs the worker to listen on a defined port. So I had to install express and just make it listen on the process.env.PORT.
My worker.js file mainly consists of Agenda.js code, which schedules and runs jobs, which is also defined in the same file.
Since I have long running jobs which are CPU intensive and also blocks event loop, this will result in much better performance for the users, since the work is not run in the same process as the main Meteor process.
The worker.js file is a standalone node.js app. It does not contain any Meteor related code. So if you need to use mongo, you have to include mongo and use it thereafter. You could use a DDP connection with the main Meteor app, though, but I did not need that.
This is a bit custom, but doesn’t requre much more work than setting up Meteor Jobs for instance. Agenda.js has very similar syntax to most of these queue plugins.
So far I can see, this is not achieved in other plugins offered, like Steve Jobs, Meteor Jobs etc. Correct me if I’m wrong.
any job that would require high levels of CPU usage. Could be generating a report that can’t be done entirely on the DB, audio/image/video processing, etc.
How about a non-meteor packages? Like node-cron for instance. Why not tap into the power of the JS ecosystem unless such a meteor package deals with an intrinsically unique Meteor related problem?
