Restart Meteor Server automatically if it crashes


#1

I am getting “Unexpected error” on my Web Page which is based on Meteor+Angular, since I am unable to debug that issue right now, so I need my server to be restarted automatically if its crashed.

How can I achieve this behavior?


#2

Our Meteor webapps operate within a CentOS 7 Linux environment and are launched by systemd.

We use systemd’s watchdog mode together with an NPM package named sd-notify to allow systemd to kill and restart our Meteor webapps if they ever lock up.

How does it work?

Systemd supports a protocol named sd_notify that allows any app (or service) to:

  1. Notify systemd when it has completed its startup initialization.

    That is, systemd will no longer presume that the app has started simply because its process is running. It will wait for the startup notification message to be received, otherwise it will assume that the app has failed to start.

  2. Send periodic watchdog (keepalive) messages to systemd to indicate that the app is still alive.

    If systemd does not receive a keepalive message within the configured timeout period, systemd will assume that your app has failed and attempt to gracefully restart it. If that fails, systemd will forcibly kill the app (i.e. kill -9) before restarting it.

Systemd watchdog mode is enabled by setting Type=notify in the [Service] section of the .service file. Its behavour is affected by these parameters:

WatchdogSec=

This configures the watchdog timeout for a service. Its value is in seconds. The watchdog is activated when the start-up is completed.
If the time between two such calls is larger than the configured time, then the service is placed in a failed state

TimeoutSec=

This configures the time in seconds that systemd will wait for the app to start-up or stop. If it doesn’t start up within the specified time, Systemd will consider the app to have failed. If it doesn’t stop within this specified time, systems will send the kill signal.

Here is an example systemd meteor.service configuration file to launch our Meteor webapp, in which the watchdog interval is 5 seconds and the startup/stop timeout is 10 seconds:

[Unit]
Description=Acme Meteor Webapp
After=network.target mongodb.service mysqld.service

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
WatchdogSec=5
TimeoutStartSec=10
Restart=always
LimitNOFILE=65535

Environment=ROOT_URL=https://acmemeteorhost.com.au/
Environment=MONGO_URL=mongodb://acme:mongopassword@localhost/acme
Environment=UNIX_SOCKET_PATH=/tmp/meteor.sock
#If your Meteor Webapp is listening on a TCP port, use the following
Environment=PORT=3000

#Set this to 0 if Meteor is running standalone
#Set this to 1 if Meteor is running behind a single proxy e.g. Nginx or Cloudflare
#Set this to 2 is Meteor is running behind two proxies, e.g. Nginx AND Cloudflare
Environment=HTTP_FORWARDED_COUNT=1

#Values taken from Meteor's settings.json
Environment='METEOR_SETTINGS={  "public" : {    "env": "Production" },  "private": { } }'

#Useful to have more libuv worker threads available
Environment=UV_THREADPOOL_SIZE=64

#If you want to import other environment variables
#EnvironmentFile=-/etc/sysconfig/acme

User=nginx
Group=nginx

WorkingDirectory=/var/www/meteor/acme/bundle
ExecStart=/usr/bin/node main.js

How do I enable my Meteor webapp to send sd_notify messages to systemd?

By installing the NPM package sd-notify, which is a Linux-specific NPM add-on written in C++ code that uses the Linux socket API.

As it is Linux-specific, you must install the NPM as an optional dependency so that if your app is launched under a non-Linux platform like Windows or OSX, the npm tool will not attempt to compile it. Here is the command:

meteor npm install --save-optional sd-notify

You will need to have Linux development tools installed as well as the systemd development libraries to successfully build the sd-notify NPM. Here’s how to install them:

If running under RHEL or CentOS:

yum groupinstall "Development Tools"
yum install systemd-devel

If running under Ubuntu:

sudo apt-get install build-essential
sudo apt-get install libsystemd-dev

Why isn’t there a native Node.js version?

It is not possible to implement systemd notification support natively in javascript under Node.js because systemd’s sd_notify protocol uses UNIX datagram sockets for its notification messages. Support for UNIX datagram sockets was removed in version 0.6 of Node.js in 2011.

For more information see: https://www.pluralsight.com/tech-blog/systemd-notify-nodejs

OK I have installed sd-notify. What do I need to add to my Meteor code?

Include this snippet in a server-side Meteor.startup() function:

 try {
  import notify from 'sd-notify';
  console.log("Notifying Systemd of service startup");
  notify.ready();

  const watchdogInterval = notify.watchdogInterval()
  if (watchdogInterval > 0) {
    console.log("Systemd watchdog interval is " + watchdogInterval + "ms");
    const interval = Math.floor(watchdogInterval / 2);
    console.log("Starting Systemd watchdog mode");
    notify.startWatchdogMode(interval);
  }
} catch (err) {
  console.log(err);
}

notify.ready() tells systemd that your app has successfully started up.

notify.watchdogInterval() automatically retrieves the value of WatchdogSec that you set in your systemd .service file. It is is greater than zero, the code will start a periodic timer that will send systemd the keepalive message twice every watchdog interval.

Why is the import statement in a try-catch block?

This allows your Meteor app to still work when you run it in under a non-Linux environment like Windows or OSX that do not support sd-notify. The import statement will generate an exception when it fails to import the missing sd-notify, which NPM did not attempt to build. The exception will be safely caught and you will see a harmless error message which you can safely ignored.

Hope this has been of assistance.