Graceful shutdown in Meteor

help
1

Hi,

I need to be sure no server-side functions are running when I restart the app.

What is the best approach to do that please ?

1 Like
2

Just from the node.js side of things you’d want to listen to the SIGTERM event using the node.js process module/api. Here is a blog post for you where most things relevant to that are being explained:

http://joseoncode.com/2014/07/21/graceful-shutdown-in-node-dot-js/

I cannot comment on the Meteor side of things, i.e. if Meteor already listens to that event and does something with it, but I think not, so you should be fine to add your own handler and then implement some form of “let current things run to the end, don’t accept any new requests”. That means you would probably have to get a hold of Meteor’s http server instance and call close() on it and maybe that would lead to the server’s (graceful) termination, but it may also be that there is some shutdown() call you need to make to Meteor or so, so it also shuts down its MongoDB connection and whatever else long-running it has going on!

HTH

3

Thank you for your reply, I did post this thread because the following methods don’t work server-side:

process.on("SIGTERM", function() {
  console.log('SIGTERM kill');
});

process.on("SIGINT", function() {
  console.log('SIGINT');
});

Any idea please ?

4

OK, I’m not familiar enough with the Meteor source to figure out what’s happening here and how to make this work. Maybe someone else will comment on that.

Other idea for you: What’s your use case, i.e. what problem are you trying to solve (that you suggest will be solved) by restarting the app while no server-side functions are running? Asking this because there might be other approaches to solving your problem.

5

Here is my use-case: customers consume an API served by a Meteor app, this API does server-side computations and inserts datas in DB. I need to be sure all requests are served before shutingdown the API to avoid having corrupted datas in the DB.

Graceful shutdown are CRITICAL for every tech claiming to be “production ready”.

6

If you know the child process then making sure they gracefully shutdown should be pretty simple.

child_process.on("close", function(code, signal){
 // switch on the signal 
 switch(signal) {
    case "SIGHUP":
          //stuff
          break;
    case "SIGTERM":
         //stuff
         break 
  }
});

Here is a working example of getting the signal.

if (Meteor.isServer) {
  Meteor.startup(function () {
    // code to run on server at startup
    spawn = Npm.require('child_process').spawn,
            grep = spawn('grep', ['ssh']);


    grep.on('close', function (code, signal) {
      console.log('child process terminated due to receipt of signal '+signal);
    });

    // send SIGHUP to process
    grep.kill('SIGHUP');
  });
}
7

Child processes are one thing, but hooking into the main process itself receiving the signals is something different. I’ve just tried to get it to work once again, just using process.on('SIGTERM', ...) etc but even with a properly deployed Meteor app the handler is not called.

@pierre I think what people usually do is relegate this task of graceful shutdown to the load balancer / frontend server layer, aka nginx, Apache or whatever one might use, in conjunction with restart scripts that first spawn a new instance of the app, add that to the load balancing config, remove the old app from the LB config, and then after some time send SIGTERM to the old app instance. And so in your case it should work with either a reasonable wait timeout after sending the last request and/or a bit of custom “ask the old instance whether there is still work being done” handling and/or using a custom way to send a termination signal if you need to do any cleanup etc.

But I do agree that it would be nice to create and/or document a simple way to hook into the Meteor shutdown procedure. I think noone has so far bothered because most shutdown / restart use cases for Meteor webapps are easily solved by using a simple LB rotation with a long enough timeout for allowing longer-running requests/work to complete. And this should be standard anyway, at least in actual production-quality deployments.

8

@seeekr yes using a rolling restart helps preventing hard shutdown of unfinished tasks. However it’s not an easy script to write, and many events can ask a process to shutdown regardless of your rolling script (cloud platform server reboot for security patches etc…).

Is there someone from Meteor staff who can tell us what is the right way to achieve that please ?

1 Like
9

Here is an example of what is required to be production-ready: hapi.js server.stop()

2 Likes
10

indeed that would be the most desirable effect!

To stop accepting new connections but finish the current ones, then shutdown

:blush: :blush: :blush: