Process Clustering? Pm2 or /arunoda/meteor-cluster?

deployment
1

How is everyone handling parallelization for their webserver processes?

Is anyone already using pm2 or meteor-cluster? How well do they work?

I can see benefits to both tools, so I’m having a hard time deciding. Our current environment is already load balanced the old fashioned way but we’re still using Forever for our process management. PM2 feels like a nice upgrade that gets us clustering for free, but I’m not sure how well it works.

2

I think you are referring to multi-core support right?

PM2 uses NodeJS cluster module for clustering. Which is not work for sticky sessions and hence with Meteor.
meteorhacks:cluster uses a different approach and it does sticky session support and works pretty much well with Meteor.

And Node’s 0.10.x cluster does not equally distribute the load. But meteor cluster do.
(both uses some low level socket sharing API in node)

3

I’m using pm2 on my dev server to play with it

I installed nvm to manage node versioning because you need a node v0.10.36 for Meteor

curl https://raw.githubusercontent.com/creationix/nvm/v0.24.0/install.sh | bash
nvm install 0.10.36
nvm use 0.10.36

Go to the meteor app folder then build the meteor app. This will create a bundle directory.

meteor build --directory /home/meteor

install some node packages required by meteor

cd /home/meteor/bundle
npm install fibers underscore source-map-support semver

start pm2

pm2 start meteor.json

or restart it

pm2 restart all

here the meteor.json:

{
  "apps": [    {
      "name": "Instance1",
      "script": "/home/meteor/bundle/main.js",
      "exec_mode": "fork_mode",
      "env": {
        "PORT": 3000,
        "MONGO_URL": "mongodb://localhost:27017/meteor",
        "ROOT_URL": "https://your.app.tld/"
      }
    },
    {
      "name": "Instance2",
      "script": "/home/meteor/bundle/main.js",
      "exec_mode": "fork_mode",
      "env": {
        "PORT": 3001,
        "MONGO_URL": "mongodb://localhost:27017/meteor",
        "ROOT_URL": "https://your.app.tld/"
      }
    }
  ]
}

The 2 meteor instances are handled by nginx with load balancing. Here some bits of the nginx config.

upstream meteorBackend  {
  server your.app.tld:3000;
  server your.app.tld:3001;
}
....

    location / {
        proxy_pass http://meteorBackend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade; # allow websockets
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Forwarded-For $remote_addr; # preserve client IP
	proxy_set_header X-Forward-Proto https;
	proxy_set_header Host $host:$server_port;

    }
...

May be not the best config for prod but it works for dev.

4

Hi @Sparser, about your question…did you made up your mind? Because i’m having the same doubt.

Thanks!

5

Yep, I went with the meteorhacks cluster module.

I’m currently second guessing my choice however as stress testing revealed
we’re still mostly processing on a single core.

6

@Sparser did you see the meteorhacks:cluster docs where you can specify probabilities that the node will accept the request? Are they all non-zero and close to one? Did you solve why it seems to be processing on a single core? If so, what was the issue?

thanks

7

I didn’t actually see that but I will go review. I left the cluster
settings at default since I didn’t see any reason to go beyond the normal
use case.

I haven’t “solved” the issue as of yet, I’m currently trying to use Kadira
as a tool to help me identify bottlenecks. I’m currently reworking some of
my subscriptions to drastically reduce the data they send over.

8

yes; also use the fields specifier left and right in your queries to
limit unnecessary data being sent.

to your first comment: I see, well I’m sure specifying that information
will help you. You could delegate a processor to route requests or carry
out serving based on a probabilistic weighting factor. I’m looking forward
to playing with that package myself.

9

I’m using Phusion Passenger and am blissfully happy with it.

10

Woah. I have never seen Phusion Passenger before. Looks impressive. How
hard was it to setup?

11

First time took me about an hour reading through the complete docs (without
a glitch).

Second time was more like 2-3 minutes.

I use the nginx integration mode, but there’s also an apache and a
standalone version.

The beauty is, it is not a server per se. It is rather a "process manager"
which shines in auto scaling.

12

Yeah, it really seems to offer some nice features, plus the option to pay
for support if we want to.

I think we’ll pilot this and see how it goes. Thanks for the reference!

1 Like
13

I don’t know whether this was there before, but it looks like Phusion Passenger created in-depth deployment instructions for a Meteor app, which is really nice.

1 Like