Hi everyone.
I have a specific question about Meteor, Galaxy, and how the containers are configured.
I have a Galaxy container that is a “Quad (pro)” sized container.
It is running a single Meteor app.
From inside the container, the Node.js process reports that there are 8 CPUs.
Running “ps” says that these are the only 3 processes running:
Now from what I can see, there is precisely 1 node process running, but I have 8 cores.
Does Meteor use the Node.js “cluster” module to leverage those additional CPUs?
Or are those cores being leveraged by way of Node.js “Worker Threads” under the covers that I can’t see?
I really want to know if all 8 of those cores are being used, or if I’m only using 1 core and paying for 8?
Yes i would like to know this too, as im looking to create some multithreaded workers using nathanschwarz:meteor-cluster to do some heavy calculations on the backend, I would like to know how the galaxy cores scale. thanks
@aon108 … I found this interesting (particularly the conclusion at the end): Benchmark: Node.js v10-v14+ Worker Threads are slow to communicate with, but keep the main loop responsive — Jakso … even if Meteor is trying to make use of the other cores with workers, it may or may actually be a win depending on what they’re trying pass around. I think the cluster module native to Node.js is really the best way for a straightforward site as it just lets multiple node processes share a listen-socket. I’ve used it before, it’s easy to set up and use, and you know you’re getting full use of all your cores.
@joeh thanks for the link it was an interesting read! In the conclusion the article mentions:
Only those very rare operations which are computationally expensive, but operate on simple input and output data, are good candidates for performing in worker threads
This explanation fits to the type of work im trying to offlay to the worker threads. a process that runs about 6seconds and effectively just loops over arrays with 1000x objects to perform mathematical calculations. which needs to be performed on about 15 different datasets that do not depend on each other so could be done all at the same time.
You could probably use them then to leverage the other cores by doing that yourself, irrespective of whether Meteor using Workers (and how it’s using them if it is).
I’m quite certain that Meteor is not using the Node.js cluster module, because I’d see multiple processes with “ps” if it were. So that means if it’s leveraging those other cores, it’s got to be doing it by using Workers, or possibly by imported modules (modules that are either compiled and use native threads or modules that use Workers). But even it’s doing that, we don’t know what it’s doing or how, so it’s hard to tell if they’re being utilized efficiently. I understand that increasing container sizes really does boost throughput similar to how you’d expect, so I guess it’s happening somehow, but I’d still like to know exactly how.