Run heavy script in background using worker threads in Meteor

help
1

I am using Meteorjs

I have one heavy query which I need to run in background

e.g:

Meteor.users.update({}, {$set:{status: 1}});

I need this query to run in background.

For that I used worker threads.

I created a worker (when my server launches in config.js):

Meteor.startup(function () {

    workerThread  = Npm.require('worker_threads');
    worker        = new workerThread.Worker('worker.js');
});

Contents of worker.js:

Meteor.startup(function () {
    
    Meteor.users.update({}, {$set:{status: 1}});
});

The above code gives me an error:

ReferenceError: Meteor is not defined

Does anyone have idea how can I run a background task in Meteorjs?

2

I’m not familiar with the worker_threads npm library, but it almost certainly runs outside of the Meteor context, so that’s why Meteor is undefined. I don’t know what to suggest here.

Anyway, Meteor.users is a reference to a MongoDB collection, so the update call actually runs on the database server. If it takes awhile for the call to return that’s just Meteor waiting for the database command to finish. You can have this be an async call if need be.

FWIW I use Steve Jobs for background tasks, although in my case they’re rather lightweight. I assume they run on the same Meteor server thread but I’m not sure.

3

Can I suggest some things?

a) first of all, I’d check if the status isn’t 1 already. A read is much quicker than a write, so avoid writes at all costs

b) secondly, use updateMany instead

So it becomes:

Meteor.users.updateMany({ status: { $ne: 1} }, { $set: {status: 1} });

Also consider using an index if it still doesn’t run fast enough. Anything that takes more than milliseconds for a query is a badly written query. Use MongoDb’s own tool with the “explain” feature to see why the query runs slow and then try to optimize it.

On backend, everything is about speed :wink:

4

This is just an example, my real script is way more complex and has way much processing, I showed this just for an example, I need to run this task in background

5

This is just an example, the real script has way much complexity and processing, there are many queries that runs one by one to get data, so I need this to run in background, so it doesnt effect my main method call performance

6

My advice is a general one that applies for every query you run against the Backend.

But you seem to know it all, so why you’re asking?

I could give you some more advice/suggestions on how to solve long running (and optimized) queries but it seems it’s not well perceived.

PS: don’t write two posts with the same content, it’s also not good practice

7

its like there is a main call that expects a response, but in between that there is a heavy recursive task running, which is causing a delay in response, so I need to run that task in background

8

We prefer using jobs to handle queues, and reactivity if the client expects an update.

For worker threads, you need a couple of things to make it Meteor-aware.

  1. Create an api endpoint using webapp
  2. Create a worker that calls the api endponit

Not pretty.

Or divide the work in async functions and use Promise.all() to call them in parallel.