Memory usage and sessions quadruple after upgrading to Meteor 3

help
21

Following up on my earlier message about inspecting meteorhacks:cluster, I went ahead and ported it to Meteor 3.

TL;DR

I now have a Meteor 3 compatible fork of meteorhacks:cluster, published on Atmosphere as dupontbertrand:cluster.

Repo: GitHub - dupontbertrand/cluster: Clustering solution for Meteor with load balancing and service discovery · GitHub

Atmosphere: The trusted source for JavaScript packages, Meteor.js resources and tools | Atmosphere (meteor add dupontbertrand:cluster)

This is a compatibility fork. I kept the same overall architecture and public API. The goal was to make it usable again on Meteor 3, not to redesign it from scratch.

What I found

The package is about 3200 lines across 15 files, and it basically does three things:

  1. Multi-core

forks child processes with per-worker ports

  1. Service discovery

registers instances in MongoDB and keeps them alive with periodic pings

  1. DDP balancing / proxying

hooks into the HTTP / WebSocket server layer, then routes requests using cookies and URL rewriting

After going through the code, the main hard blocker for Meteor 3 was the MongoDB discovery backend. It relied heavily on Meteor.wrapAsync around an embedded MongoDB 1.4.x driver, which obviously does not fit well in Meteor 3 anymore.

The rest was mostly smaller compatibility / maintenance work.

What I changed

  • rewrote the discovery backend with async/await

  • updated the embedded MongoDB usage to mongodb@6.12.0

  • made Cluster.connect() and Cluster.register() work correctly in Meteor 3 startup flow

  • replaced underscore usage with native ES features

  • updated npm dependencies (cookies, http-proxy, portscanner)

  • fixed the Buffer() deprecation

  • fixed a pre-existing IPC listener issue

  • fixed and adapted parts of the test suite for Meteor 3

  • made the balancer more transport-aware so it plays better with the pluggable transport work in #14231

I also documented the work here:

Changelog / notes: cluster/METEOR3-PORT.md at master · dupontbertrand/cluster · GitHub

What I did not change

I intentionally did not try to redesign the package.

So at this stage:

  • the public API stays the same

  • the worker pool architecture stays the same

  • the OverShadowServerEvent approach is still there

That part is still invasive and a bit fragile, but I wanted to keep the scope focused and first see whether a compatibility port was actually viable.

Positioning

I do not see this as a new recommended scaling architecture for Meteor.

If all you need is multi-core, PM2 or another external process manager is much simpler, for example:


pm2 start main.js -i max

But for apps that were actually relying on meteorhacks:cluster features like service auto-registration, DDP-aware inter-instance proxying, or Cluster.discoverConnection(), this fork may provide a practical migration path to Meteor 3.

Current status

  • tested on Linux with Meteor 3.4 and Node 22

  • multi-core, discovery, and service registration are working in my tests

  • no Windows testing yet

  • published on Atmosphere as dupontbertrand:cluster@2.0.0

Repo again: GitHub - dupontbertrand/cluster: Clustering solution for Meteor with load balancing and service discovery · GitHub

If anyone here was blocked on Meteor 3 because of meteorhacks:cluster, I’d be very interested to know whether this fork works for your setup.

@tpmh31292 — since you mentioned you already patched meteorhacks:cluster locally for Meteor 3, I’d also be curious to compare notes if you’re up for it.

22

Wow, that is so nice!

Maybe we can bring it to Meteor after it becomes stable. IMO, Meteor misses a native cluster feature since we’ve been working hard on performance last months.

What do you think @nachocodoner?

23

Yeah, that would be a nice addition.

Though, we should likely keep it in the community scope for now, so people can try it and give feedback. @dupontbertrand, you could likely open a PR to add this package to the community package docs, so people are aware of it from the official docs.

As discussed here, there are also simpler ways to get multi-core capabilities, like PM2 + Nginx, instead of an app-specific approach such as meteorhacks:cluster. But for apps that relied on the package-specific features, having this Meteor 3-compatible fork available is still a good path while people evaluate what works best.

Documenting it in the meantime would help us not lose what already exists, get feedback from real users, and make a better call later on whether any approach should influence the core.

24

During peak times, I used 4 servers with Meteor Up, each having dual cores, but my app only utilized a single core. This library looks like it could solve my problem—awesome! :tada:

25

@welkinwong Glad if it can help! Feel free to send me feedback here or on the GitHub repo anytime, there is still plenty of room for improvement :slight_smile:

26

This is excellent, thanks heaps.
What I did was only updated dependencies in a quick attempt to make it run with Meteor 3.4 to fix the production app.
Will try out pm2 also as that multi-core is all we need at the moment.

27

Done !