Memory usage and sessions quadruple after upgrading to Meteor 3

[dupontbertrand]

Following up on my earlier message about inspecting meteorhacks:cluster, I went ahead and ported it to Meteor 3.

TL;DR

I now have a Meteor 3 compatible fork of meteorhacks:cluster, published on Atmosphere as dupontbertrand:cluster.

Repo: GitHub - dupontbertrand/cluster: Clustering solution for Meteor with load balancing and service discovery · GitHub

Atmosphere: The trusted source for JavaScript packages, Meteor.js resources and tools | Atmosphere (meteor add dupontbertrand:cluster)

This is a compatibility fork. I kept the same overall architecture and public API. The goal was to make it usable again on Meteor 3, not to redesign it from scratch.

What I found

The package is about 3200 lines across 15 files, and it basically does three things:

1. Multi-core

forks child processes with per-worker ports

2. Service discovery

registers instances in MongoDB and keeps them alive with periodic pings

3. DDP balancing / proxying

hooks into the HTTP / WebSocket server layer, then routes requests using cookies and URL rewriting

After going through the code, the main hard blocker for Meteor 3 was the MongoDB discovery backend. It relied heavily on Meteor.wrapAsync around an embedded MongoDB 1.4.x driver, which obviously does not fit well in Meteor 3 anymore.

The rest was mostly smaller compatibility / maintenance work.

What I changed

• rewrote the discovery backend with async/await

• updated the embedded MongoDB usage to mongodb@6.12.0

• made Cluster.connect() and Cluster.register() work correctly in Meteor 3 startup flow

• replaced underscore usage with native ES features

• updated npm dependencies (cookies, http-proxy, portscanner)

• fixed the Buffer() deprecation

• fixed a pre-existing IPC listener issue

• fixed and adapted parts of the test suite for Meteor 3

• made the balancer more transport-aware so it plays better with the pluggable transport work in #14231

I also documented the work here:

Changelog / notes: https://github.com/dupontbertrand/cluster/blob/master/METEOR3-PORT.md

What I did not change

I intentionally did not try to redesign the package.

So at this stage:

• the public API stays the same

• the worker pool architecture stays the same

• the OverShadowServerEvent approach is still there

That part is still invasive and a bit fragile, but I wanted to keep the scope focused and first see whether a compatibility port was actually viable.

Positioning

I do not see this as a new recommended scaling architecture for Meteor.

If all you need is multi-core, PM2 or another external process manager is much simpler, for example:

pm2 start main.js -i max

But for apps that were actually relying on meteorhacks:cluster features like service auto-registration, DDP-aware inter-instance proxying, or Cluster.discoverConnection(), this fork may provide a practical migration path to Meteor 3.

Current status

• tested on Linux with Meteor 3.4 and Node 22

• multi-core, discovery, and service registration are working in my tests

• no Windows testing yet

• published on Atmosphere as dupontbertrand:cluster@2.0.0

Repo again: GitHub - dupontbertrand/cluster: Clustering solution for Meteor with load balancing and service discovery · GitHub

If anyone here was blocked on Meteor 3 because of meteorhacks:cluster, I’d be very interested to know whether this fork works for your setup.

@tpmh31292 — since you mentioned you already patched meteorhacks:cluster locally for Meteor 3, I’d also be curious to compare notes if you’re up for it.

[italojs]

Wow, that is so nice!

Maybe we can bring it to Meteor after it becomes stable. IMO, Meteor misses a native cluster feature since we’ve been working hard on performance last months.

What do you think @nachocodoner?

[nachocodoner]

Yeah, that would be a nice addition.

Though, we should likely keep it in the community scope for now, so people can try it and give feedback. @dupontbertrand, you could likely open a PR to add this package to the community package docs, so people are aware of it from the official docs.

As discussed here, there are also simpler ways to get multi-core capabilities, like PM2 + Nginx, instead of an app-specific approach such as meteorhacks:cluster. But for apps that relied on the package-specific features, having this Meteor 3-compatible fork available is still a good path while people evaluate what works best.

Documenting it in the meantime would help us not lose what already exists, get feedback from real users, and make a better call later on whether any approach should influence the core.

[welkinwong]

During peak times, I used 4 servers with Meteor Up, each having dual cores, but my app only utilized a single core. This library looks like it could solve my problem—awesome!

[dupontbertrand]

@welkinwong Glad if it can help! Feel free to send me feedback here or on the GitHub repo anytime, there is still plenty of room for improvement

[tpmh31292]

This is excellent, thanks heaps.
What I did was only updated dependencies in a quick attempt to make it run with Meteor 3.4 to fix the production app.
Will try out pm2 also as that multi-core is all we need at the moment.

[dupontbertrand]

Done !