How to fix ERR_EMPTY_RESPONSE for my url


#1

We have a ubuntu server. I build my meteor app and run it with pm2. Recently I have a strange behavior concerning the url I put my app on.

I have this domain backtocart.co on godaddy. And I have this subdomain - app.backtocart.co

Whenever I go to that subdomain I get browser response

app.backtocart.co didn’t send any data. ERR_EMPTY_RESPONSE

However in my app inside server I have a redirect from/ route to /login. And that redirect happens.
On the other hand if I go to my app by the public ip http://217.182.200.46:3001 it just works great!

So I am not even sure why is this problem happening. Because of the app or some server configs?

Here are the dns configs from godaddy.

enter image description here

As you can see the root points to a different ip than the subdomain. On 145.239.24.196 I have nginx server.chatservice.backtocart.co is on nginx as well however it works fine I get nginx response.
I tried to put the app on this domain as well. There I just get a blank page from meteor.

My nginx configs seem to be fine as well. Here are the configs

#proxy_cache_path  /home/html_catch  levels=1:2 keys_zone=appbacktocartcocatch:2m max_size=50m;
#proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
#proxy_cache_valid 200 302 10m;
#proxy_cache_valid 404 1m;


upstream appbacktocartco {
#        server localhost:3000;
	server localhost:3001;
}


server {
     listen 145.239.24.196:80;
     server_name app.backtocart.co;

     location / {

       proxy_cache appbacktocartcocatch;

       proxy_set_header   Host $host;
       proxy_set_header   X-Real-IP $remote_addr;
       proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header   X-Forwarded-Proto $scheme;
       proxy_http_version 1.1;
       proxy_set_header   Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_pass         http://appbacktocartco;
       proxy_read_timeout 90;
    }


    listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/app.backtocart.co/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/app.backtocart.co/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    } # managed by Certbot

}

And here is the general nginx config

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
	worker_connections 768;
         multi_accept on;
}

http {
	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	include /etc/nginx/mime.types;
	default_type application/octet-stream;
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;
	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;
	gzip on;
	gzip_disable "msie6";
	gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;


        proxy_cache_path  /home/html_catch  levels=1:2 keys_zone=appbacktocartcocatch:2m max_size=50m;
        proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
        proxy_cache_valid 200 302 10m;
        proxy_cache_valid 404 1m;




	include /etc/nginx/sites-enabled/*.conf;




}

Hiring a devops/meteor developer
#2

any error message logged in /var/log/nginx/error.log?


#3

Here is what catches the eye

2018/04/25 13:04:17 [emerg] 5518#5518: unknown directive "l" in /etc/nginx/sites-enabled/app.backtocart.co.conf:8
2018/04/25 13:04:30 [emerg] 5548#5548: unknown directive "l" in /etc/nginx/sites-enabled/app.backtocart.co.conf:8
2018/04/25 13:05:10 [alert] 5627#5627: *1 write() to "/var/log/nginx/access.log" failed (28: No space left on device) while logging request, client: 83.26.242.86, server: app.backtocart.co, request: "GET /websocket HTTP/1.1", upstream: "http://127.0.0.1:3000/websocket", host: "app.backtocart.co"
2018/04/25 13:05:10 [error] 5627#5627: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 64.213.69.2, server: app.backtocart.co, request: "GET /websocket HTTP/1.1", upstream: "http://[::1]:3000/websocket", host: "app.backtocart.co"
2018/04/25 13:05:18 [crit] 5627#5627: *32 pwritev() "/var/lib/nginx/proxy/1/00/0000000001" failed (28: No space left on device) while reading upstream, client: 185.44.229.105, server: app.backtocart.co, request: "GET /dashboard HTTP/1.1", upstream: "http://127.0.0.1:3000/dashboard", host: "app.backtocart.co"
2018/04/25 13:05:21 [crit] 5627#5627: *50 pwritev() "/var/lib/nginx/proxy/2/00/0000000002" failed (28: No space left on device) while reading upstream, client: 185.44.229.105, server: app.backtocart.co, request: "GET /dashboard HTTP/1.1", upstream: "http://127.0.0.1:3000/dashboard", host: "app.backtocart.co"
2018/04/25 13:05:21 [error] 5627#5627: *54 connect() failed (111: Connection refused) while connecting to upstream, client: 24.20.120.197, server: app.backtocart.co, request: "GET /websocket HTTP/1.1", upstream: "http://[::1]:3000/websocket", host: "app.backtocart.co"
2018/04/25 13:05:23 [crit] 5627#5627: *53 pwritev() "/var/lib/nginx/proxy/3/00/0000000003" failed (28: No space left on device) while reading upstream, client: 185.44.229.105, server: app.backtocart.co, request: "GET /dashboard HTTP/1.1", upstream: "http://127.0.0.1:3000/dashboard", host: "app.backtocart.co"
2018/04/25 13:05:32 [error] 5627#5627: *77 connect() failed (111: Connection refused) while connecting to upstream, client: 141.136.91.12, server: app.backtocart.co, request: "POST /createNewVisit HTTP/1.1", upstream: "http://[::1]:3000/createNewVisit", host: "app.backtocart.co", referrer: "https://www.incredo.co/contact"
2018/04/25 13:06:01 [error] 5627#5627: *89 connect() failed (111: Connection refused) while connecting to upstream, client: 141.136.91.12, server: app.backtocart.co, request: "POST /setDisconnectedAt HTTP/1.1", upstream: "http://[::1]:3000/setDisconnectedAt", host: "app.backtocart.co", referrer: "https://www.incredo.co/"
2018/04/25 13:06:21 [alert] 5627#5627: *90 write() to "/var/log/nginx/access.log" failed (28: No space left on device) while logging request, client: 141.136.91.12, server: app.backtocart.co, request: "OPTIONS /createNewVisit HTTP/1.1", upstream: "http://127.0.0.1:3000/createNewVisit", host: "app.backtocart.co"
2018/04/25 13:06:21 [error] 5627#5627: *89 connect() failed (111: Connection refused) while connecting to upstream, client: 141.136.91.12, server: app.backtocart.co, request: "OPTIONS /getProjectIdByUrl HTTP/1.1", upstream: "http://[::1]:3000/getProjectIdByUrl", host: "app.backtocart.co"
2018/04/25 13:06:40 [crit] 5627#5627: *141 pwritev() "/var/lib/nginx/proxy/4/00/0000000004" failed (28: No space left on device) while reading upstream, client: 185.44.229.105, server: app.backtocart.co, request: "GET /login HTTP/1.1", upstream: "http://127.0.0.1:3000/login", host: "app.backtocart.co"
2018/04/25 13:06:40 [crit] 5627#5627: *142 pwritev() "/var/lib/nginx/proxy/5/00/0000000005" failed (28: No space left on device) while reading upstream, client: 185.44.229.105, server: app.backtocart.co, request: "GET /login HTTP/1.1", upstream: "http://127.0.0.1:3000/login", host: "app.backtocart.co"
2018/04/25 13:06:59 [error] 5627#5627: *146 connect() failed (111: Connection refused) while connecting to upstream, client: 91.93.38.58, server: app.backtocart.co, request: "GET /websocket HTTP/1.1", upstream: "http://[::1]:3000/websocket", host: "ap2018/04/25 13:58:17 [notice] 9430#9430: signal process started
2018/04/25 13:58:22 [notice] 9440#9440: signal process started
2018/04/25 13:58:24 [notice] 9450#9450: signal process started
2018/04/25 13:58:40 [notice] 9460#9460: signal process started
2018/04/25 13:59:25 [alert] 9266#9266: *1 open socket #14 left in connection 3
2018/04/25 13:59:25 [alert] 9266#9266: *2 open socket #16 left in connection 4

It seems the server is out of disk space?

if I do df -h in bash here is what I get

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        20G   19G     0 100% /
devtmpfs        7.9G     0  7.9G   0% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           7.9G  822M  7.1G  11% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/md1        487M   22M  436M   5% /boot
/dev/md3        200G  7.1G  183G   4% /home

But I am not sure what all the numbers mean.
Does root seems to be full but I still have 183GB free on disk?


#4

So yes, the problem was that there was no available space on root. Clearing up space on root solved the issue.