Concurrent workers with Meteor collections

announce
1

Hey,

Recently I’ve came up with a really simple pattern to implement concurrent workers using all the benefits that Meteor collections and observers brings to us. It’s not my first attempt I must admit and I only decided to share the solution I’ve found and ask for feedback because I was finally able to achieve what I wanted in ~50 lines of vanilla Meteor code. It’s available here:

In short, we want to implement a delayed tasks queue with multiple workers in a way that the following three properties are fulfilled:

  1. Each task will be performed by only one worker.
  2. No more that a constant number of tasks is performed by a single worker in parallel.
  3. Ideally there should be some kind of fallback scenario, e.g. when worker crashes while task is being processed.

A problem like this is traditionally solved with dedicated software, but for someone who wants to keep things simple (no additional dependencies), who does not expect a huge workers load and only wants the properties 1-3, it might be tempting to try and have a simple mongo-based solution.

I am quite sure that whoever tried to implement it “by hand” would agree with me that the hardest part is to make sure that the tasks queue on each worker is properly balanced. Well, maybe it’s just me, but at first it was not clear at all that maintaining the queue of the right size can be easily done with the following observer:

Tasks.find({
  $or: [
    { workerId: null },
    { workerId },
  ],
}, {
  limit: 10,
  // With "workerId: -1" nulls come last, which is exactly what we want.
  sort: { workerId: -1 },
}).observe({
  added:   processTask,
  changed: processTask,
});

I think that it may even look trivial for someone who just briefly looks at the code, but let me just point out that the tricky part here is the $or selector. You will see why if you look at the complete implementation.

Since this solution feels a little bit too simple for the problem which it is supposed to solve, I was wondering if there’s a gotcha here which I cannot see right away. So if there’s anyone who already tried this and it turned out not to work so great at the end of the day, please let me know :slight_smile:

Anyway, how do you like it?

1 Like