Memory usage and sessions quadruple after upgrading to Meteor 3

help
7

I’m using Meteor 3.4

8

Do you use any third-party atmosphere package on your app? publish-composite, redis-oplog, reactive-aggregate or any other data or jobs related package?

It would be nice to see the packages that you use for your app.

9

In the first screenshot, I noticed that the number of sessions also increased at the same time you switched to Meteor 3. It makes it difficult to know if the changes in metrics are related to the Meteor release, or to the number of sessions. I do notice, though, that in Meteor 2, the memory usage was related to the number of sessions - when the sessions decreased, the memory decreased. However, with Meteor 3.4, the memory doesn’t significantly decrease when the sessions decrease. You could be right that there is a memory leak.

For the high cpu usage, it does seem related to the number of sessions: it does go down when the sessions go down. If you are using continuous cpu profiling, you can look at what was causing the high cpu usage.

I’m working on some new tools to help with memory leaks. If you send me a message with the app you are monitoring, I can give you early access.

1 Like
10

@nachocodoner yes I do use publish-composite and redis-oplog and strigops:job-collection

Here is my packages file

cultofcoders:redis-oplog
disable-oplog@1.0.8
meteor-base@1.5.2             # Packages every Meteor app needs to have
mobile-experience@1.1.2       # Packages for a great mobile UX
mongo@2.2.0                   # The database Meteor supports right now
reactive-var@1.0.13            # Reactive variable for tracker

standard-minifier-css@1.10.0   # CSS minifier run for production mode
standard-minifier-js@3.2.0    # JS minifier run for production mode
es5-shim@4.8.1                # ECMAScript 5 compatibility for older browsers
ecmascript@0.17.0              # Enable ECMAScript2015+ syntax in app code
typescript@5.9.3              # Enable TypeScript syntax in .ts and .tsx modules
shell-server@0.7.0            # Server-side component of the `meteor shell` command
hot-module-replacement@0.5.4  # Update client in development without reloading the page

static-html@1.5.0             # Define static page content in .html files
react-meteor-data       # React higher-order component for reactively tracking Meteor data
accounts-password@3.2.2
ccorcos:subs-cache
montiapm:agent
aldeed:simple-schema
reywood:publish-composite
ferjep:persistent-session
session@1.2.2
strigops:job-collection
montiapm:profiler
11

Thanks for your input, I sent you a message.

12

it’s a interesting insight since I got a similar behavior testing oplog vs change streams, getting OOM while stress oplog
can you simulate this behavior outside production? so we can test 3.5-beta and confirm the possible memory leak is in oplog module(or not)

13

I think I got it:

Meteor 2 — collection.js:481 :

sub.onStop(function() {
  observeHandle.stop();  // SYNC — Fibers block until completion
});

Meteor 3 — collection.js:133 :

sub.onStop(async function() {
  return await observeHandle.stop();  // ASYNC — return Promise
});

But _callStopCallbacks is not update :

_callStopCallbacks: function () {
  var callbacks = self._stopCallbacks;
  self._stopCallbacks = [];
  callbacks.forEach(function (callback) {
    callback();  // ← Promise return but ignored
  });
}

Let me test and I’ll open a PR if it’s the problem

14

The PR :+1:

3 Likes
15

Thanks for looking into this everyone, so I actually found out the reason why Memory and CPU usage were so high.
Turns out in Meteor 2, I had meteorhacks:cluster so the app utilizing all the cores on our servers.
While upgrading to Meteor 3, I removed it because of the package constraint with webapp makes it incompatible with Meteor 3.
Cloned meteorhacks:cluster locally and updated its package.js to make it works with Meteor 3 and now metrics are looking good again.

16

Yes, when using third-party packages it’s good to, if you experience really weird behavior in consumption, incrementally silence all those extra parts. That can help you find the culprit once you see the behavior stop happening (or appearing! just like in your use case).

In normal apps, we have been monitoring this and the experience you reported did not happen to us, so with high probability the reason was an extra package causing those bad behaviors, which is why we asked about it, or a very edge case in the core that we are not aware of, and could we fix for the core. Apparently we got one. Even with a different cause, it’s nice to explore and fix other inconsistencies.

But this is still very generic but worth for anybody running into these issues. In your case, as you described, that package had been removed and the effect was the opposite. It was needed for load balancing as the purpose of the package and had to exist to balance the metrics back on previous versions.

meteorhacks:cluster could be resurrected (11 years ago last release!) and migrated to Meteor 3.

You mentioned that you had to update it in a cloned copy, at least to get minimum compatibility. Would you like to publish that version so others can use it too?

17

I think it’s way better to use something like pm2 (one-line pm2 start main.js -i max --name "meteor-app" and configure nginx to use sticky sessions :thinking:

1 Like
18

Good suggestion for the multi-core case. But looking at the meteorhacks:cluster package (not really experienced with it though), PM2 + Nginx might not cover everything teams were relying on like:

  • Auto-registration when adding or removing instances
  • Any instance being able to act as a balancer, avoiding a single point of failure
  • DDP-aware inter-instance proxying
  • Cluster.discoverConnection() for microservice setups

If any of those were in use, the migration effort could be non-trivial for a production app. There could be production apps that used meteorhacks:cluster purely for multi-core and never touched the other features, in which case PM2 would be a fine replacement. A simpler, less overengineered approach may be enough, and that is true. But that doesn’t mean the migration will be easy, or that metric behavior in their app won’t be off during the transition until they fully understand the new setup vs meteorhacks:cluster’s specifics.

Having a minimally compatible Meteor 3 version as a first step, while teams assess what they actually need, seems like the safer path in short term.

1 Like
19

You’re right, I was mostly looking at the multi-core angle

For apps that only used meteorhacks:cluster to spread load across CPU cores, PM2 + Nginx is probably enough. But for apps that depended on the package-specific features like auto-registration, DDP-aware proxying, or discoverConnection(), that is a different story indeed :grimacing:

My main concern is that the package appears to do fairly invasive monkey-patching of DDP and WebApp internals. In Meteor 3, with the async changes, that feels like the kind of thing that can easily become fragile across upgrades

So I could see value in a minimally compatible Meteor 3 version as a migration path, I can inspect the code a bit and report back on how realistic that port would be :+1:

2 Likes
20

They were at least able to get the package working in their app again, with the metrics showing properly too, likely as an effect of proper load balancing again on their app. So they may have reached the minimum compatibility needed.

There could still be other edges that need attention, so any extra input would be great, just in case there are more fixes to cover. :rocket:

21

Following up on my earlier message about inspecting meteorhacks:cluster, I went ahead and ported it to Meteor 3.

TL;DR

I now have a Meteor 3 compatible fork of meteorhacks:cluster, published on Atmosphere as dupontbertrand:cluster.

Repo: GitHub - dupontbertrand/cluster: Clustering solution for Meteor with load balancing and service discovery · GitHub

Atmosphere: The trusted source for JavaScript packages, Meteor.js resources and tools | Atmosphere (meteor add dupontbertrand:cluster)

This is a compatibility fork. I kept the same overall architecture and public API. The goal was to make it usable again on Meteor 3, not to redesign it from scratch.

What I found

The package is about 3200 lines across 15 files, and it basically does three things:

  1. Multi-core

forks child processes with per-worker ports

  1. Service discovery

registers instances in MongoDB and keeps them alive with periodic pings

  1. DDP balancing / proxying

hooks into the HTTP / WebSocket server layer, then routes requests using cookies and URL rewriting

After going through the code, the main hard blocker for Meteor 3 was the MongoDB discovery backend. It relied heavily on Meteor.wrapAsync around an embedded MongoDB 1.4.x driver, which obviously does not fit well in Meteor 3 anymore.

The rest was mostly smaller compatibility / maintenance work.

What I changed

  • rewrote the discovery backend with async/await

  • updated the embedded MongoDB usage to mongodb@6.12.0

  • made Cluster.connect() and Cluster.register() work correctly in Meteor 3 startup flow

  • replaced underscore usage with native ES features

  • updated npm dependencies (cookies, http-proxy, portscanner)

  • fixed the Buffer() deprecation

  • fixed a pre-existing IPC listener issue

  • fixed and adapted parts of the test suite for Meteor 3

  • made the balancer more transport-aware so it plays better with the pluggable transport work in #14231

I also documented the work here:

Changelog / notes: https://github.com/dupontbertrand/cluster/blob/master/METEOR3-PORT.md

What I did not change

I intentionally did not try to redesign the package.

So at this stage:

  • the public API stays the same

  • the worker pool architecture stays the same

  • the OverShadowServerEvent approach is still there

That part is still invasive and a bit fragile, but I wanted to keep the scope focused and first see whether a compatibility port was actually viable.

Positioning

I do not see this as a new recommended scaling architecture for Meteor.

If all you need is multi-core, PM2 or another external process manager is much simpler, for example:


pm2 start main.js -i max

But for apps that were actually relying on meteorhacks:cluster features like service auto-registration, DDP-aware inter-instance proxying, or Cluster.discoverConnection(), this fork may provide a practical migration path to Meteor 3.

Current status

  • tested on Linux with Meteor 3.4 and Node 22

  • multi-core, discovery, and service registration are working in my tests

  • no Windows testing yet

  • published on Atmosphere as dupontbertrand:cluster@2.0.0

Repo again: GitHub - dupontbertrand/cluster: Clustering solution for Meteor with load balancing and service discovery · GitHub

If anyone here was blocked on Meteor 3 because of meteorhacks:cluster, I’d be very interested to know whether this fork works for your setup.

@tpmh31292 — since you mentioned you already patched meteorhacks:cluster locally for Meteor 3, I’d also be curious to compare notes if you’re up for it.

5 Likes
22

Wow, that is so nice!

Maybe we can bring it to Meteor after it becomes stable. IMO, Meteor misses a native cluster feature since we’ve been working hard on performance last months.

What do you think @nachocodoner?

3 Likes
23

Yeah, that would be a nice addition.

Though, we should likely keep it in the community scope for now, so people can try it and give feedback. @dupontbertrand, you could likely open a PR to add this package to the community package docs, so people are aware of it from the official docs.

As discussed here, there are also simpler ways to get multi-core capabilities, like PM2 + Nginx, instead of an app-specific approach such as meteorhacks:cluster. But for apps that relied on the package-specific features, having this Meteor 3-compatible fork available is still a good path while people evaluate what works best.

Documenting it in the meantime would help us not lose what already exists, get feedback from real users, and make a better call later on whether any approach should influence the core.

4 Likes
24

During peak times, I used 4 servers with Meteor Up, each having dual cores, but my app only utilized a single core. This library looks like it could solve my problem—awesome! :tada:

4 Likes
25

@welkinwong Glad if it can help! Feel free to send me feedback here or on the GitHub repo anytime, there is still plenty of room for improvement :slight_smile:

2 Likes
26

This is excellent, thanks heaps.
What I did was only updated dependencies in a quick attempt to make it run with Meteor 3.4 to fix the production app.
Will try out pm2 also as that multi-core is all we need at the moment.

2 Likes