RFC: First-class job queue for Meteor (PR #14286)

core
1

I’ve opened a PR adding a built-in job queue to core: https://github.com/meteor/meteor/pull/14286

The short version: Meteor doesn’t have a first-class answer for background jobs. Rails has Active Job, Laravel has Queues, Next.js folks reach for Inngest or Trigger.dev. Meteor developers end up picking between aging community packages or rolling their own. This PR adds meteor/jobs (the queue) and meteor/worker-pool (optional CPU offloading via worker threads), both built on top of meteor/thread-context from #14281.

Before this gets too far along I want feedback from the community on the design — especially the API surface, since that’s the thing that’s hardest to change later.

Key design decisions I’d like input on:

  • MongoDB-backed, no Redis. Leader election, job storage, and reactive pickup all use MongoDB (oplog tailing for low-latency claim, atomic ops for leader election – could be change streams for 3.5+). No external coordination service. Trade-off: it’s one less thing to deploy, but it does put more load on Mongo.
  • Global string-keyed registry. Jobs are registered by name (Jobs.register({ name: 'sendEmail', run })) and enqueued by name (Jobs.run('sendEmail', data)). I went this way to avoid forcing imports across the codebase, similar to how Meteor.methods works. Grubba already raised whether returning a handle from register would be better — I’d love more opinions here.
  • Worker thread offloading is opt-in per job. Add offload: true and the handler runs in a worker thread with proxied access back to collections and methods via thread-context. Most jobs probably shouldn’t offload; this is for CPU-heavy work that would block the event loop.
  • Built-in primitives instead of separate packages for each feature. Retries, deduplication, cancellation (with AbortSignal), cron scheduling, run-and-wait, lifecycle events, and monitoring publications all ship in the one package. Open question: is this too much surface area for core, or is the right call to keep it cohesive? I like the idea of having all the tools a developer needs right in the core package, like other frameworks I mentioned above.
  • Test modes baked in. inline runs synchronously with no DB for unit tests; manual enqueues but only runs when you explicitly trigger. Wanted to make this testable from day one rather than as an afterthought.
  • Three publications for monitoring (jobs.status, jobs.history, jobs.job), gated by an authorize callback. The idea is admin dashboards are built by the app author rather than shipping a UI in core. I am hesitant on this feature – perhaps this should be delegated to app implementation.

I’ll publish all three packages to Atmosphere this week so people can kick the tires without building from source. Happy to answer questions about anything in the diff (it’s ~13k lines, so I don’t blame anyone for not reading the whole thing).

Specifically curious about:

  1. Anyone strongly for or against the string-name registration vs. returned-handle pattern?
  2. Anything obviously missing that you’d want before adopting this?
  3. Concerns about MongoDB-only coordination at scale?
1 Like