I’ve got some long-running tasks (10 - 20 minutes) that of course, block the main thread. Is there an official Meteor way or package, or alternate approach, to move them to a service worker, or otherwise prevent them from blocking?
I don’t know what type of “blocking” you’re talking about. But I have a case that there’s a request would take 1 or 2 minutes. On the client, I just make a Fire and Forget request, then subscribe to a publication which will update when the request is done.
By designing like that, the user can do other stuffs while waiting for the request finished.
2 Likes
By blocking, I mean that the home page can’t be loaded while that task is running. I call this task with async await, so I thought that meant I was off the hook in terms of blocking the main thread. But I guess in the async function, I call a lot of LLM endpoints and they can be very slow to respond. Could that be blocking the main thread?
I searched the docs for “Fire and Forget” but nothing came up. I found a StackOverflow post talking about it but it was out of date, and referenced fibers. Will “Fire and Forget” work for a task that takes 10-20 minutes, and if so, is there a link where I can read about it?
Claude is telling me to set it up to run as a child process, and then launch it via a cron job manager. It provides this sample code:
Setting Up Child Process
#!/usr/bin/env node
/**
* Standalone worker process for long-running tasks
* This runs in a completely separate Node.js process
*/
const { startAIAgent } = require('../report_latest_cybersecurity_news');
const args = process.argv.slice(2);
const gatherArticles = args.includes('--gather');
const generateReports = args.includes('--reports');
async function main() {
console.log('🔧 Worker process started');
console.log(` PID: ${process.pid}`);
console.log(` Gather: ${gatherArticles}, Reports: ${generateReports}`);
try {
await startAIAgent(gatherArticles, generateReports);
console.log('✅ Worker process completed successfully');
process.exit(0);
} catch (error) {
console.error('❌ Worker process failed:', error);
process.exit(1);
}
}
main();
Launching from a cron job
import { Queue, Worker } from 'bullmq';
import { spawn } from 'child_process';
import path from 'path';
import IORedis from 'ioredis';
const REDIS_URL = process.env.REDIS_URL || 'redis://localhost:6379';
const connection = new IORedis(REDIS_URL, {
maxRetriesPerRequest: null,
enableReadyCheck: false,
});
const newsGatheringQueue = new Queue('cybersecurity-news-gathering', { connection });
// Worker that spawns child processes
const newsGatheringWorker = new Worker(
'cybersecurity-news-gathering',
async (job) => {
const { name, taskData } = job.data;
const startTime = new Date();
console.log(`\n${'='.repeat(60)}`);
console.log(`▶️ [WORKER] Starting job: ${name}`);
console.log(` Job ID: ${job.id}`);
console.log(` Main process PID: ${process.pid}`);
console.log(`${'='.repeat(60)}\n`);
// Spawn the worker as a separate process
const workerPath = path.join(__dirname, 'worker_process.js');
const args = [];
if (taskData.createReport) args.push('--gather');
if (taskData.startupRun) args.push('--reports');
return new Promise((resolve, reject) => {
const childProcess = spawn('node', [workerPath, ...args], {
stdio: 'inherit', // Show child process output in main console
env: process.env, // Pass environment variables
});
childProcess.on('exit', (code) => {
const endTime = new Date();
const duration = ((endTime - startTime) / 1000).toFixed(2);
if (code === 0) {
console.log(`\n${'='.repeat(60)}`);
console.log(`✅ [WORKER] Job completed successfully`);
console.log(` Duration: ${duration}s`);
console.log(` Child PID: ${childProcess.pid}`);
console.log(`${'='.repeat(60)}\n`);
resolve({ success: true, duration, pid: childProcess.pid });
} else {
console.error(`\n${'='.repeat(60)}`);
console.error(`❌ [WORKER] Job failed with exit code ${code}`);
console.error(` Duration: ${duration}s`);
console.error(`${'='.repeat(60)}\n`);
reject(new Error(`Worker process exited with code ${code}`));
}
});
childProcess.on('error', (error) => {
console.error('❌ Failed to start child process:', error);
reject(error);
});
});
},
{
connection,
concurrency: 2, // Can run 2 child processes simultaneously
}
);
newsGatheringWorker.on('completed', (job) => {
console.log(`📊 [BullMQ] Job ${job.id} completed`);
});
newsGatheringWorker.on('failed', (job, err) => {
console.error(`📊 [BullMQ] Job ${job?.id} failed:`, err.message);
});
export async function initializeScheduledJobs() {
console.log('\n📅 Initializing BullMQ with child process workers...');
await newsGatheringQueue.add(
'Gather Cybersecurity News Articles',
{
name: 'Gather Cybersecurity News Articles',
taskData: {
createReport: true,
startupRun: false,
},
},
{
repeat: {
pattern: '15 20 * * *',
tz: 'America/New_York',
},
}
);
console.log(' ✔ Jobs will run in separate Node.js processes');
console.log(' ✔ Main server remains responsive');
console.log(' ✔ Concurrency: 2 processes can run simultaneously\n');
}
export { newsGatheringQueue, newsGatheringWorker, connection };
“Fire and forget” in async world is you call an async function without await keyword. It’s just my way, not the industry standard though.
Interesting – okay I will keep that in mind as I research this.
If something takes 10-20 minutes it looks like something of a process which might be better in a queue system. There are multiple for Meteor.
That way you also get retry and monitoring logic.
You might want to use the new AbortController() API, and here’s why it’s better for general use than a simple await-less call:
- Error Handling: It encapsulates all error logic (network issues, API status errors, and timeouts) within one unit.
- Timeouts: It gracefully handles operations that hang indefinitely, preventing resource leaks or a poor user experience.
- Clear Intent: When the caller uses
await fireAndForget(), they know exactly what they are getting: the result or a specific error message (like “Request timed out”) within a reasonable timeframe.
But it is not “better” if your goal is truly to run something in a non-blocking, fire-and-forget manner where you do not care about the result or potential failure.
It’s in the name: Fire and forget. I don’t care if it works or how long it will take because I subscribe to a publication which will tell me if it has timeout, error or success with result.
It might be better using a queue system like @lucfranken said above, but in some cases you just want to make it simple.
If you don’t care if it works, you can probably just delete those lines of code.
Fire-and-forget doesn’t mean that you don’t care about it. It means that you don’t need the result from it.
If I run it in a queue system, will the code run on a Web Worker, i.e. a child thread?
Relevant to this thread – this appeared on HackerNews today –
It seems to be working now – I can run the tasks on service worker threads and they don’t block the main thread. The biggest part of doing it was to get the service workers to have access to the relevant Meteor settings, since the Meteor object didn’t seem to be available to the service workers. I just had to copy the relevant settings from Meteor.settings, to the server process.env, and make sure the code accessed those variables from that location. I’m using:
import worker from 'node:worker_threads
…to implement the service worker threads.