Using DDP.connect as a task queue

help
1

I’m sending a task to a worker on a different url using this format:

Workers = function (workers) {
  var self = this;
  this.workers = [];

  workers.forEach(function (url) {
    var worker = DDP.connect(url);

    Tracker.autorun(function () {
      var status = worker.status();
      if (status.connected) {
        console.log("worker connected: " + url);
        self.workers.push(worker);
      } else {
        var index = self.workers.indexOf(worker);
        if (index >= 0) {
          console.log("worker disconnected: " + url);
          self.workers.splice(index, 1);
        }
      }
    });
  });

  this._nextWorker = 0;
}

var workers = new Workers(process.env.WORKER_URLS.split(';'));

Meteor.methods({
  cj: function(task) {
    this.unblock();
    return workers.cj(task);
  },

If I send a really long task, however, the worker disconnects. As soon as the task has finished on the worker, the worker reconnects, and the main app tries to send the task again. It gets stuck in infinite loops.

I’ve tried every combination of unblock and defer I can think of! Can I stop the worker from being disconnected? Or do I have to alter the behaviour of the workers in some way?

Thanks!

2

What do you mean by task disconnects. May be that’s a CPU heavy task Meteor node’s eventloop is blocking.

Then Meteor can’t send heartbeats to that workers and disconnects. I hope that’s happening here. If so, there is no way to do prevent the disconnection.

I mention this in bulletproof meteor as a generic way to do this.
You may need to build some functionality around it.

But, if you don’t wanna do it. Try using this package: https://atmospherejs.com/vsivsi/job-collection

3

Brilliant, thank you. Yes, it’s a CPU intensive task that is running inside a node package so your suggestion seems very likely. I wanted to check there wasn’t anything obvious before I built more functionality around it. Easy enough to check if the task is still running I guess! And the package looks very interesting.