Should Galaxy apps use cluster?

deployment
1

Should Galaxy apps use meteorhack’s cluster, or do they need it in order to take advantage of multiple cores?

2

I’m going to say no. Galaxy is designed and built by the super-smart folks at Meteor and I would bet they have it fine-tuned to only use one-core per machine. And if more cores are leveraged they probably know how to utilize them behind the scenes.

They wrote all their own back-end tools using AWS to specifically handle Meteor apps, sticky-sessions, and all the specifics required by Meteor apps. More info here if you’re curious:

I haven’t heard of anyone using cluster with Galaxy. And in fact, I think cluster may not be maintained anymore. So proceed with caution.

P.S. There’s also a critical bug that causes crashes when using cluster in multi-core mode. It’s an open issue on the cluster github page so it hasn’t been fixed. It’s stuff like this that’s causing me to shy away from cluster or at least use it in single-core mode.

1 Like
3

Great informative answer, thanks!

4

There’s also this open issue with cluster on GitHub talking about Galaxy not supporting it:

5

@evolross is right, MDG have designed the container sizes for optimal CPU utilisation.

From Galaxy support:

In the machines we use on Amazon, 1 Quad sized instance on Galaxy translates into 1 full CPU core. 1 double sized instance gets 0.5 of a CPU core on AWS.

We use m4.4xlarge AWS machines, which have 64 GB RAM and 16 vCPU. So a Quad sized container gives you 4 GB and 1 vcpu. Accordingly, one Double sized container gives you 2 GB and 0.5 vcpu.

1 Like
6

I’m fairly sure that issue isn’t related to multi-core support.

7

No, this is not best practice. Cluster should be not used in production. It attempts to abstract a server functionality into meteor, which is not what meteor was built for … meteor is not a web server.

Read more at the bottom of this page: https://github.com/ramezrafla/meteor-deployment

The way to do it is to have a meteor instance per core, then a load balancer directs traffic to each instance as needed (see the nginx conf in that same repo).