Memory usage and sessions quadruple after upgrading to Meteor 3

help
21

Following up on my earlier message about inspecting meteorhacks:cluster, I went ahead and ported it to Meteor 3.

TL;DR

I now have a Meteor 3 compatible fork of meteorhacks:cluster, published on Atmosphere as dupontbertrand:cluster.

Repo: GitHub - dupontbertrand/cluster: Clustering solution for Meteor with load balancing and service discovery · GitHub

Atmosphere: The trusted source for JavaScript packages, Meteor.js resources and tools | Atmosphere (meteor add dupontbertrand:cluster)

This is a compatibility fork. I kept the same overall architecture and public API. The goal was to make it usable again on Meteor 3, not to redesign it from scratch.

What I found

The package is about 3200 lines across 15 files, and it basically does three things:

  1. Multi-core

forks child processes with per-worker ports

  1. Service discovery

registers instances in MongoDB and keeps them alive with periodic pings

  1. DDP balancing / proxying

hooks into the HTTP / WebSocket server layer, then routes requests using cookies and URL rewriting

After going through the code, the main hard blocker for Meteor 3 was the MongoDB discovery backend. It relied heavily on Meteor.wrapAsync around an embedded MongoDB 1.4.x driver, which obviously does not fit well in Meteor 3 anymore.

The rest was mostly smaller compatibility / maintenance work.

What I changed

  • rewrote the discovery backend with async/await

  • updated the embedded MongoDB usage to mongodb@6.12.0

  • made Cluster.connect() and Cluster.register() work correctly in Meteor 3 startup flow

  • replaced underscore usage with native ES features

  • updated npm dependencies (cookies, http-proxy, portscanner)

  • fixed the Buffer() deprecation

  • fixed a pre-existing IPC listener issue

  • fixed and adapted parts of the test suite for Meteor 3

  • made the balancer more transport-aware so it plays better with the pluggable transport work in #14231

I also documented the work here:

Changelog / notes: cluster/METEOR3-PORT.md at master · dupontbertrand/cluster · GitHub

What I did not change

I intentionally did not try to redesign the package.

So at this stage:

  • the public API stays the same

  • the worker pool architecture stays the same

  • the OverShadowServerEvent approach is still there

That part is still invasive and a bit fragile, but I wanted to keep the scope focused and first see whether a compatibility port was actually viable.

Positioning

I do not see this as a new recommended scaling architecture for Meteor.

If all you need is multi-core, PM2 or another external process manager is much simpler, for example:


pm2 start main.js -i max

But for apps that were actually relying on meteorhacks:cluster features like service auto-registration, DDP-aware inter-instance proxying, or Cluster.discoverConnection(), this fork may provide a practical migration path to Meteor 3.

Current status

  • tested on Linux with Meteor 3.4 and Node 22

  • multi-core, discovery, and service registration are working in my tests

  • no Windows testing yet

  • published on Atmosphere as dupontbertrand:cluster@2.0.0

Repo again: GitHub - dupontbertrand/cluster: Clustering solution for Meteor with load balancing and service discovery · GitHub

If anyone here was blocked on Meteor 3 because of meteorhacks:cluster, I’d be very interested to know whether this fork works for your setup.

@tpmh31292 — since you mentioned you already patched meteorhacks:cluster locally for Meteor 3, I’d also be curious to compare notes if you’re up for it.