I’ve got some long-running tasks (10 - 20 minutes) that of course, block the main thread. Is there an official Meteor way or package, or alternate approach, to move them to a service worker, or otherwise prevent them from blocking?
I don’t know what type of “blocking” you’re talking about. But I have a case that there’s a request would take 1 or 2 minutes. On the client, I just make a Fire and Forget request, then subscribe to a publication which will update when the request is done.
By designing like that, the user can do other stuffs while waiting for the request finished.
1 Like
By blocking, I mean that the home page can’t be loaded while that task is running. I call this task with async await, so I thought that meant I was off the hook in terms of blocking the main thread. But I guess in the async function, I call a lot of LLM endpoints and they can be very slow to respond. Could that be blocking the main thread?
I searched the docs for “Fire and Forget” but nothing came up. I found a StackOverflow post talking about it but it was out of date, and referenced fibers. Will “Fire and Forget” work for a task that takes 10-20 minutes, and if so, is there a link where I can read about it?
Claude is telling me to set it up to run as a child process, and then launch it via a cron job manager. It provides this sample code:
Setting Up Child Process
#!/usr/bin/env node
/**
* Standalone worker process for long-running tasks
* This runs in a completely separate Node.js process
*/
const { startAIAgent } = require('../report_latest_cybersecurity_news');
const args = process.argv.slice(2);
const gatherArticles = args.includes('--gather');
const generateReports = args.includes('--reports');
async function main() {
console.log('🔧 Worker process started');
console.log(` PID: ${process.pid}`);
console.log(` Gather: ${gatherArticles}, Reports: ${generateReports}`);
try {
await startAIAgent(gatherArticles, generateReports);
console.log('✅ Worker process completed successfully');
process.exit(0);
} catch (error) {
console.error('❌ Worker process failed:', error);
process.exit(1);
}
}
main();
Launching from a cron job
import { Queue, Worker } from 'bullmq';
import { spawn } from 'child_process';
import path from 'path';
import IORedis from 'ioredis';
const REDIS_URL = process.env.REDIS_URL || 'redis://localhost:6379';
const connection = new IORedis(REDIS_URL, {
maxRetriesPerRequest: null,
enableReadyCheck: false,
});
const newsGatheringQueue = new Queue('cybersecurity-news-gathering', { connection });
// Worker that spawns child processes
const newsGatheringWorker = new Worker(
'cybersecurity-news-gathering',
async (job) => {
const { name, taskData } = job.data;
const startTime = new Date();
console.log(`\n${'='.repeat(60)}`);
console.log(`▶️ [WORKER] Starting job: ${name}`);
console.log(` Job ID: ${job.id}`);
console.log(` Main process PID: ${process.pid}`);
console.log(`${'='.repeat(60)}\n`);
// Spawn the worker as a separate process
const workerPath = path.join(__dirname, 'worker_process.js');
const args = [];
if (taskData.createReport) args.push('--gather');
if (taskData.startupRun) args.push('--reports');
return new Promise((resolve, reject) => {
const childProcess = spawn('node', [workerPath, ...args], {
stdio: 'inherit', // Show child process output in main console
env: process.env, // Pass environment variables
});
childProcess.on('exit', (code) => {
const endTime = new Date();
const duration = ((endTime - startTime) / 1000).toFixed(2);
if (code === 0) {
console.log(`\n${'='.repeat(60)}`);
console.log(`✅ [WORKER] Job completed successfully`);
console.log(` Duration: ${duration}s`);
console.log(` Child PID: ${childProcess.pid}`);
console.log(`${'='.repeat(60)}\n`);
resolve({ success: true, duration, pid: childProcess.pid });
} else {
console.error(`\n${'='.repeat(60)}`);
console.error(`❌ [WORKER] Job failed with exit code ${code}`);
console.error(` Duration: ${duration}s`);
console.error(`${'='.repeat(60)}\n`);
reject(new Error(`Worker process exited with code ${code}`));
}
});
childProcess.on('error', (error) => {
console.error('❌ Failed to start child process:', error);
reject(error);
});
});
},
{
connection,
concurrency: 2, // Can run 2 child processes simultaneously
}
);
newsGatheringWorker.on('completed', (job) => {
console.log(`📊 [BullMQ] Job ${job.id} completed`);
});
newsGatheringWorker.on('failed', (job, err) => {
console.error(`📊 [BullMQ] Job ${job?.id} failed:`, err.message);
});
export async function initializeScheduledJobs() {
console.log('\n📅 Initializing BullMQ with child process workers...');
await newsGatheringQueue.add(
'Gather Cybersecurity News Articles',
{
name: 'Gather Cybersecurity News Articles',
taskData: {
createReport: true,
startupRun: false,
},
},
{
repeat: {
pattern: '15 20 * * *',
tz: 'America/New_York',
},
}
);
console.log(' ✔ Jobs will run in separate Node.js processes');
console.log(' ✔ Main server remains responsive');
console.log(' ✔ Concurrency: 2 processes can run simultaneously\n');
}
export { newsGatheringQueue, newsGatheringWorker, connection };
“Fire and forget” in async world is you call an async function without await keyword. It’s just my way, not the industry standard though.
Interesting – okay I will keep that in mind as I research this.