Trapping SIGINT/SIGTERM doesn't seem to work in meteor

help
1

I’m trying to do graceful shutdowns as my application receives SIGINT or SIGTERM. I basically do this:

    setupSignalHandlers = () => {
        process.on("SIGTERM", () => {
            console.log("SIGTERM received");
            this.stop();
        });
        process.on("SIGINT", () => {
            console.log("SIGINT received");
            this.stop();
        });
    };

And have my cleanup code in this.stop(). If I start this using node myFile.js, it works just fine. If I run it in a meteor process, my callbacks are not triggered by the signals. I can’t find any examples of a meteor specific way of accomplishing this either. Does anyone have any similar experiences, or have successfully accomplished something to the same effect as what I’m trying?

1 Like
2

Did you try wrapping the callback with Meteor.bindEnvironment()

3

As in

...
process.on("SIGTERM", Meteor.bindEnvironment(() => { /* intercept logic */ }));

?

I tried now with no result. But also wasn’t aware of the bindEnvironment method, so I might not have set it up correctly.

4

I think this might be of interest here:

5

I’ll give it a go, but looks as if that lib is also relying on process.on(signal) to function.

6

The package might also be in need of an update @grubba @fredmaiaarantes

2 Likes
7

I’ve used the ddp-graceful-shutdown package in many projects before without any issues. Its code is pretty small, so @jlugner might want to take a look at its repository.

I just installed it in a simple project to test it on Galaxy, and it worked!

I’m using the code below that I added in a new file named shutdown-handler.js. The purpose of the package is to shut the DDP connections down gradually, so I’m overriding installSIGTERMHandler() so I can do something before they are closed.

import { Meteor } from 'meteor/meteor';
import { DDPGracefulShutdown } from '@meteorjs/ddp-graceful-shutdown';

class SimpleDDPGracefulShutdown extends DDPGracefulShutdown {
  installSIGTERMHandler() {
    process.on(
      'SIGTERM',
      Meteor.bindEnvironment(() => {
        const gracePeriod =
          process.env.METEOR_SIGTERM_GRACE_PERIOD_SECONDS || 30;

        console.log(
          `Received SIGTERM. Shutting down in ${gracePeriod} seconds.`
        );

        this.closeConnections({ log: true });
      })
    );
  }
}

new SimpleDDPGracefulShutdown({
  gracePeriodMillis: 1000 * process.env.METEOR_SIGTERM_GRACE_PERIOD_SECONDS,
  server: Meteor.server,
}).installSIGTERMHandler();

I’m importing it in main.js, which is my server entry point. And that’s it.

import { Meteor } from 'meteor/meteor';
import { Migrations } from 'meteor/percolate:migrations';

import './shutdown-handler'; //Here
import './db/migrations';
import './tasks/tasks.publications';
import './tasks/tasks.methods';

/**
 * This is the server-side entry point
 */
Meteor.startup(() => {
  Migrations.migrateTo('latest');
});

When my container is being stopped, I can see the message below on Galaxy logs:

Screenshot 2023-11-14 at 17.40.59
Screenshot 2023-11-14 at 17.40.591792×378 98.8 KB

My current grace period is 40 seconds:

Screenshot 2023-11-14 at 17.47.45
Screenshot 2023-11-14 at 17.47.452038×496 27.4 KB

2 Likes
8

I can’t seem to replicate… Every now and then I reach the callbacks (~10% of the time?), but even in those cases it immediately exits. I’m trying to use it in my background worker. Idea is to let it finish any running jobs when a signal is received, but not start any new. This is virtually what it’s doing, but with sleeps in place of any actual job logic:

import { DDPGracefulShutdown } from "@meteorjs/ddp-graceful-shutdown";
const SLEEP_TIME = 100;

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

export class Worker extends DDPGracefulShutdown {
    running = false;
    grabFirstQueuedJob = async () => {
        console.log("Grabbing first queued job");
        await sleep(1000);
        console.log("Pretending to grab first and running it");
    };

    executeJob = async () => {
        console.log("Executing job");
        await sleep(10000);
        console.log("Done executing job");
    };

    run = async () => {
        if (!this.running) {
            console.log("Worker stopped running");
            return;
        }

        try {
            await this.poll();
            setTimeout(this.run, SLEEP_TIME);
        } catch (e) {
            console.error("An error occurred while polling:", e);
            setTimeout(this.run, SLEEP_TIME);
        }
    };

    stop = () => {
        console.log("Stopping worker");
        this.running = false;
    };

    setupSignalHandlers = () => {
        process.on(
            "SIGTERM",
            Meteor.bindEnvironment(() => {
                console.log("SIGTERM received");
                this.stop();
                this.closeConnections({ log: true });
            }),
        );
        process.on(
            "SIGINT",
            Meteor.bindEnvironment(() => {
                console.log("SIGINT received");
                this.stop();
                this.closeConnections({ log: true });
            }),
        );
    };

    poll = async () => {
        await this.grabFirstQueuedJob();
        await this.executeJob();
    };

    start = async () => {
        console.log("Starting worker");
        this.setupSignalHandlers();
        this.running = true;

        await this.run();
    };
}

if (process.env.IS_WORKER) {
    const worker = new Worker({
        gracePeriodMillis: 1000 * 30,
        server: Meteor.server,
    });
    worker.start();
}

The file is loaded first thing in my startup script. I’m running everything locally

9

How are you sending SIGTERM to the process locally? Would you be able to deploy and test it on a server?

10

Yeah, locally. Sending signals using ctrl + c. Works for regular node without issues. I’ll have a go at doing it on a server (Heroku).

11

Same result I’m afraid. Callback doesn’t get called, executing stops immediately.

12

Hello @fredmaiaarantes, is this also applicable in closing lingering MongoDB connections? This is my first time encountering problems with MongoDB connection and one of the recommended solutions is to close out some lingering connections on a graceful shutdown but I do not have any knowledge to do that.

Thanks in advance

13

Hey @stevenitsme05, I haven’t used them for this purpose, but I believe you could use the ddp-graceful-shutdown package to run any process you need.

14

I’ve struggled with this quite a bit as well. I haven’t dug to much into the behaviour in production, but I believe it’s not that prominent there.

From what I can gather it seems to be related to the way the meteor-tool shuts down its child processes on server restart. Sometimes, depending on how long the process has been running, the hooks will get hit much less often as time progresses. If I remember correctly, the tool is registering and un-registering event listeners. During the un-registration process it might remove your event listeners instead of the ones added by Meteor.