Websocket failure with nginx reverse proxy

1

hi,

i have a meteor app launched with jupyterhub such that

jupyterhub --> nginx:3000 --> meteorapp:39000

now, jupyterhub will only forward traffic based on the path: such that

/user/ytl/proxy/3000 --> / --> /usr/ytl/proxy/3000

ie, i use nginx in between to map from what is passed into it from jupyterhub so that the meteor app can pas back the relevant prefix. specifically i set

ROOT_URL="http://localhost:39000/user/ytl/proxy/3000/"

and i create a nginx server like so:

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

server {
  listen 3000;

  server_name jupyter-dev;

  location / {
    proxy_pass http://localhost:39000/user/ytl/proxy/3000/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_redirect off;
    proxy_intercept_errors on;
    proxy_buffering off;
    proxy_cache_bypass $http_upgrade;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto  $scheme;
    proxy_set_header X-NginX-Proxy true;
    proxy_ssl_server_name on;
  }
}

so from this, the meteorapp itself seems to load okay. however, the websockets appears to have some issues:

15
151582×834 117 KB

it appears to do some xhr and then spits out

"{"msg":"nosub","id":"T6PignvprPwiw2z9b","error":{"isClientSafe":true,"error":500,"reason":"Internal server error","message":"Internal server error [500]","errorType":"Meteor.Error"}}"

the webapp.logs show:

Exception from sub user.state id pq7KzcfSbQxnLpfQd TypeError: this.ready is not a function
    at Subscription.Meteor.publish [as _handler] (imports/api/Users/server/publications.js:34:15)
    at currentArgumentChecker.withValue (packages/check/match.js:118:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
    at DDP._CurrentPublicationInvocation.withValue (packages/ddp-server/livedata_server.js:1043:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Subscription._runHandler (packages/ddp-server/livedata_server.js:1041:51)
    at Session._startSubscription (packages/ddp-server/livedata_server.js:859:9)
    at Session.sub (packages/ddp-server/livedata_server.js:625:12)
    at packages/ddp-server/livedata_server.js:559:43
Exception from sub user.state id xfDn2dbuXzGFtADSX TypeError: this.ready is not a function
    at Subscription.Meteor.publish [as _handler] (imports/api/Users/server/publications.js:34:15)
    at currentArgumentChecker.withValue (packages/check/match.js:118:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
    at DDP._CurrentPublicationInvocation.withValue (packages/ddp-server/livedata_server.js:1043:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Subscription._runHandler (packages/ddp-server/livedata_server.js:1041:51)
    at Session._startSubscription (packages/ddp-server/livedata_server.js:859:9)
    at Session.sub (packages/ddp-server/livedata_server.js:625:12)
    at packages/ddp-server/livedata_server.js:559:43
Exception from sub projects id jr8THcd69RhJHiyFc TypeError: Cannot read property 'name' of null
    at get_project_query (imports/api/Projects/server/publications.js:9:60)
    at Subscription.Meteor.publish [as _handler] (imports/api/Projects/server/publications.js:17:26)
    at currentArgumentChecker.withValue (packages/check/match.js:118:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
    at DDP._CurrentPublicationInvocation.withValue (packages/ddp-server/livedata_server.js:1043:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Subscription._runHandler (packages/ddp-server/livedata_server.js:1041:51)
    at Session._startSubscription (packages/ddp-server/livedata_server.js:859:9)
    at Session.sub (packages/ddp-server/livedata_server.js:625:12)
    at packages/ddp-server/livedata_server.js:559:43
Exception from sub switcher.workspaces id w9bBv36ijyB2fCGFg TypeError: Cannot read property 'name' of null
    at get_project_query (imports/api/Workspaces/server/publications.js:8:60)
    at Subscription.Meteor.publish [as _handler] (imports/api/Workspaces/server/publications.js:28:15)
    at currentArgumentChecker.withValue (packages/check/match.js:118:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
    at DDP._CurrentPublicationInvocation.withValue (packages/ddp-server/livedata_server.js:1043:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Subscription._runHandler (packages/ddp-server/livedata_server.js:1041:51)
    at Session._startSubscription (packages/ddp-server/livedata_server.js:859:9)
    at Session.sub (packages/ddp-server/livedata_server.js:625:12)
    at packages/ddp-server/livedata_server.js:559:43
Exception from sub switcher.job id jDCMDA7NgBHQRZaEP TypeError: Cannot read property 'name' of null
    at get_project_query (imports/api/Jobs/server/publications.js:8:60)
    at get_allowed_projects (imports/api/Jobs/server/publications.js:15:24)
    at Subscription.Meteor.publish [as _handler] (imports/api/Jobs/server/publications.js:59:7)
    at currentArgumentChecker.withValue (packages/check/match.js:118:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
    at DDP._CurrentPublicationInvocation.withValue (packages/ddp-server/livedata_server.js:1043:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Subscription._runHandler (packages/ddp-server/livedata_server.js:1041:51)
    at Session._startSubscription (packages/ddp-server/livedata_server.js:859:9)
    at Session.sub (packages/ddp-server/livedata_server.js:625:12)
    at packages/ddp-server/livedata_server.js:559:43
Exception from sub projects.recent id 7rNBjxmBgTWBM378L TypeError: Cannot read property 'name' of null
    at get_project_query (imports/api/Projects/server/publications.js:9:60)
    at Subscription.Meteor.publish.list [as _handler] (imports/api/Projects/server/publications.js:63:15)
    at currentArgumentChecker.withValue (packages/check/match.js:118:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
    at DDP._CurrentPublicationInvocation.withValue (packages/ddp-server/livedata_server.js:1043:15)
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12)
    at Subscription._runHandler (packages/ddp-server/livedata_server.js:1041:51)
    at Session._startSubscription (packages/ddp-server/livedata_server.js:859:9)
    at Session.sub (packages/ddp-server/livedata_server.js:625:12)
    at packages/ddp-server/livedata_server.js:559:clock430:
2

Can you please add the corresponding publication function? It may be rather a production code related error than nginx related.

3

Yeah, that would be my feeling too. Looks like a real bug in your code.

As an aside, try looking for Meteor’s fallback to polling and see if that works. Your Rube Goldberg nginx configuration is going to be very difficult to diagnose in this environment.

4

thanks guys. unfortunately, the meteor code is not mine and is ‘closed’ source. i’m not that familar with meteor: can you point me to the relevant docs to work out how i can find what publication function is being used and/or how to set it to polling? thanks!