DO, NGINX: 502 Bad Gateway persist; 90% certain its a Meteor issue!

ljanssen · May 5, 2016, 2:25am

Hi!

I have been researching and trying out many different ways to kill my 502 Bad Gateway but to no success! I am deploying my Meteor app on Digital Ocean’s $10 droplet running Ubuntu 14.04 x64

Domain Registrar:
GoDaddy.
My normal home web-page’s domain is mydomain.com which forwards to an instapage website, but I have a subdomain login.mydomain.com to redirect (301) to my DO droplet. Added an A record for login that points to the DO IP. Seems to resolve just fine except for some DMARC issues (and delegation of course due to no website returned).

Deploying Meteor
I follow the this guide https://www.digitalocean.com/community/tutorials/how-to-deploy-a-meteor-js-application-on-ubuntu-14-04-with-nginx where I run “sudo npm install” while being the myapp user.

My /etc/nginx/sites-available/myapp.conf file:

server_tokens off;
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    listen 80 default_server; 
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    server_name login.mydomain.com;
    location / {
        rewrite     ^ https://$server_name$request_uri? permanent;
    }
}
server {
    listen 443 ssl spdy;
    server_name login.mydomain.com;
    root html;
    index index.html;

    ssl_certificate /etc/nginx/ssl/server.pem;
    ssl_certificate_key /etc/nginx/ssl/server.key;

    ssl_stapling on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 5m;

    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK';

    add_header Strict-Transport-Security "max-age=31536000;";
    if ($http_user_agent ~ "MSIE" ) {
        return 303 https://browser-update.org/update.html;
    }
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade; # allow websockets
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Forwarded-For $remote_addr; # preserve client IP
        if ($uri != '/') {
            expires 30d;
        }
    }
}

My /etc/init/myapp.conf file:

start on started mongodb and runlevel [2345]
stop on shutdown
respawn
respawn limit 10 5
setuid myapp
setgid myapp

script
    export PATH=/opt/local/bin:/opt/local/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    export NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript
    export PWD=/home/myapp
    export HOME=/home/myapp
    export BIND_IP=127.0.0.1
    export PORT=8080
    export HTTP_FORWARDED_COUNT=1
    export MONGO_URL=mongodb://localhost:27017/myapp
    export ROOT_URL=https://login.mydomain.com
    exec node /home/myapp/bundle/main.js >> /home/myapp/myapp.log
end script

NOTES

I set-up SSL exactly like this: https://www.digitalocean.com/community/tutorials/how-to-create-a-ssl-certificate-on-nginx-for-ubuntu-12-04
When I run “start myapp” it says “myapp start/running, process 12218” and when I type “status myapp” it then shows “myapp stop/waiting”.
/home/myapp/myapp.log is empty so somehow the Meteor app thinks everything is okay.
“nginx -t” returns all OK except SSL stapling warning
My NGINX error log says 2016/05/04 00:30:42 [error] 18144#0: *28 connect() failed (111: Connection refused) while connecting to upstream, client: , server: login.mydomain.com, request: “GET /favicon.ico HTTP/1.1”, upstream: “http://127.0.0.1:8080/favicon.ico”, host: “login.mydomain.com”, referrer: “https://login.mydomain.com/”
My code runs on my localhost just fine on my dev laptop (Mac OS), (not tried the build version, just ran the unpacked version from my git pull)
I cannot access it on :8080 so it seems it just can’t reach Meteor on the server.
“sudo netstat -plutn” output:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 12209/nginx
tcp 0 0 127.0.0.1:28017 0.0.0.0:* LISTEN 2880/mongod
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1103/sshd
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 12209/nginx
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 2880/mongod
tcp6 0 0 :::80 :::* LISTEN 12209/nginx
tcp6 0 0 :::22 :::* LISTEN 1103/sshd

Anybody any idea why there’s still a 502 Bad Gateway issue?
I am literally pulling my hair out here for a few days.

vigorwebsolutions · May 5, 2016, 3:14am

Out of curiosity, can you spin up another droplet and try a mup/mupx deployment?

Not saying that you shouldn’t be able to deploy the way you are, but that would confirm a mis-step somewhere in setting up the droplet in my mind?

cstrat · May 5, 2016, 3:41am

Is there any reason why you’re using port 8080 and not 3000?
Could that be causing an issue on the server somewhere?
I haven’t looked at the other config info you’ve pasted in great detail, it mostly looks like what I’ve used myself except that that port…

On my server I am running three apps all on separate ports, 3000, 3100 and 3200. No issues.

ljanssen · May 5, 2016, 3:59am

Thank you for your answer. I have changed all occurences of 8080 to 3000 and it still does not work. I also tried the mongoDB port to be 3001 but that did not do it either.

ljanssen · May 5, 2016, 4:00am

It does work with mupx, fortunately. However, I do not want to be dependent on mupx and like to have ssl in my own hands with NGINX

vigorwebsolutions · May 5, 2016, 4:23am

Right on. Wasn’t suggesting mup as a replacement, just trying to troubleshoot. On a side note, you can use mup with an nginx reverse proxy so you can handle your own ssl.

Back to your problem: are you able to run something like meteor --port 8080 and access it?

ljanssen · May 5, 2016, 4:43am

Nope! the meteor command isn’t even found, I never directly installed meteor on the Digital Ocean machine.

vigorwebsolutions · May 5, 2016, 4:31pm

Again, out of curiosity, reading through the post and comments, issues, found:

ljanssen · May 5, 2016, 4:56pm

Thanks for the help! I’ve looked over that one comment. I installed the current V5.x version of nodejs after purging it, and added comments to all the export paths in my init file.
It still does not work though. But when I type ‘status myapp’ it now says it’s running which is a good change! When I run ‘nginx -t’ under the myapp user, it gives me errors. Only if I run it under root it says NGINX OK.
The myapp.log that’s defined in the init file is also missing, which is strange, since the output of main.js should be put there, so Im assuming something else is wrong too…

Ben305 · May 5, 2016, 5:24pm

I’m using mup (from kadirahq) to deploy the app to a Digital Ocean Droplet, I have created my own docker image that is used to run the app. Nginx is running as reverse-proxy and configured manually.

I would suggest using mup.

ljanssen · May 5, 2016, 10:14pm

Im not completely shying away from mup, but Im building an app that handles financial data and using enforcessl is a must, and story goes it ain’t easy with mup. Also do not like to have another intermediary step between deployment that can break when you most need it

vigorwebsolutions · May 5, 2016, 10:26pm

Is enforcessl a must, or is ssl a must? I’m all for rolling your own deployment, but my setup looks like:

Deployment: MUP
Server: NGINX
SSL: LetsEncrypt

On the qualys test, the score looks like:

I’d say that on the first walkthrough, if you aren’t familiar with mup, nginx, or le, you could expect to spend a day on it. Now, though, I’d say I could spin up a droplet and have this setup in an hour or less…

vigorwebsolutions · May 5, 2016, 10:27pm

Also, I completely get how frustrating it can be not to figure something out. And an alternative can sometimes seem like giving up. But time is precious!

prime · May 5, 2016, 11:06pm

So it’s not only me!

I had the same error like yours and was trying to figured it out all over the internet but couldn’t solved the error and then just gave up Nginx and use naked Node JS alone.

Without Nginx, Meteor app work but I encountered websocket connection issue. Again, I was trying to figured it out all over the internet but couldn’t solved the problem and just gave up websocket. I disable websocket and thinking what’s the point I included webscoket related package like DDP if I disable websocket.

ljanssen · May 5, 2016, 11:21pm

Agreed that time is precious.

ljanssen · May 5, 2016, 11:22pm

Maybe I am wrong after all. I thought that you NEED to have force-ssl package in Meteor, but now that I think of it NGINX will rewrite any incoming http into https anyway. So maybe I do not need it if I use NGINX?

vigorwebsolutions · May 5, 2016, 11:52pm

Exactly, my basic conf looks like:

server {
    listen 80;
    return 301 https://$host$request_uri;
}

server {
	listen 443 default_server;

	root /usr/share/nginx/html;
	index index.html index.htm;

	# Make site accessible from http://localhost/
	server_name localhost;


        ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
	ssl_dhparam /etc/letsencrypt/live/mydomain.com/dhparams.pem;

    	ssl on;
    	ssl_session_cache  builtin:1000  shared:SSL:10m;
    	ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

	location / {
		proxy_pass http://localhost:3333;
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection 'upgrade';
		proxy_set_header Host $host;
		proxy_cache_bypass $http_upgrade;
	}

        location ~ /.well-known {
                allow all;
        }

}

ljanssen · May 10, 2016, 8:53pm

I still haven’t found the issue. I’ve deployed it for now with MUPX and it works fine. But I really do not like the intermediary step. Also, mupx logs cannot be found. If I type in my localhost/dev machine ‘mupx logs -f’ it says it cannot find it.
Any ideas on how to setup the deployment without mupx?
If not, any ideas on how to fetch the logs? They don’t exist on the server on the supposed-to-be path

vigorwebsolutions · May 10, 2016, 9:51pm

Honestly, I’ve been using mup more than mupx, though I do have a few apps running on mupx. Also, because of the docker/container, I think fetching the mupx logs is a bit different – https://github.com/arunoda/meteor-up/issues/831?

miriam7 · May 11, 2016, 9:25am

I am using passenger phusion + nginx on my own server, which works really nicely. They have a Digital/Ocean/Nginx tutorial as well: https://www.phusionpassenger.com/library/walkthroughs/deploy/meteor/digital_ocean/nginx/oss/install_passenger_main.html Might be worth checking out.