Attach debugger to unresponsive production server

help
1

Hi all,

We have a meteor application serving ~10’000 users. From time to time, we experience the server to become unresponsive (hang / freeze). This means it does not directly return a HTTP 5xx error-code but clients can no more establish a websocket connection.

The server is deployed using ‘mup’. If the server becomes unresponsive, I can connect via ssh and show the logs. However, this is not very helpful to find the cause of a freeze.

As a workaround, I have to restart meteor (mup stop, mup start) to get everything back online. This is very unsatisfying.

What techniques do you recommend to debug such hangs? How can I connect a local debugger to productive meteor / mup environment? Is there any documentation to start with?

Best regards,

  • seb
2

Hi,

You might want to take a look at Monti APM. It lets you monitor your application performance.
https://app.montiapm.com/

And there is qualia:profile package using which you can take a dump for your cpu usage.
https://atmospherejs.com/qualia/profile

btw, which node version are you using?

3

Thanks! I’ll give that Monti APM a try - looks like what I’m looking for.

At the moment I am using Meteor 1.8.0.2 / Node 8.11.4 (I need to remove some mobile compatibility issues before I can update).

  • seb
4

To actually answer your question, you can connect a local debugger using a shell package and a remote debugging package from the npm world. It will Just Work. If you have better control the docker host that MUP manages for you, it’ll be safest to create a “Bastion Host,” like an ikev2 VPN server, that connects you to the internal docker network using only a password you know, and connect your debugger that way, rather than exposing the debugging port directly to the internet. It’s not like debugger packages support passwords.

That’s usually pretty bad and indicating of exhaustion of resources, especially bandwidth / networking and not necessarily your application.

Make sure you’re using a machine / instance that is appropriate for your usage. If you are using an AWS instance, use one with a 10 gigabit interface: https://ec2instances.info.

5

Thanks for your hints @doctorpangloss - but I’m stuck. Can you clarify:

you can connect a local debugger using a shell package and a remote debugging package from the npm world. It will Just Work.

Shouldn’t it be enough to set debug to true in mup, and then redirect the default port (5858) via SSH to the local machine?

ssh -L 5858:localhost:5858 SERVERDOMAIN

module.exports = {
...
  app: {
  ...
    // All options are optional.
    buildOptions: {
      ...
      
      // Set to true to disable minification and bundling,
      // and include debugOnly packages
      debug: true,
	}
  }
}

http://meteor-up.com/docs.html#everything-configured

But thats not working (no connections are accepted at that port) - probably because the docker image abernix/meteord does not expose the debug port. Is there an easy way to modify that docker image? (exposing port or setup the VPN server in the image) :thinking: