Trapping SIGINT/SIGTERM doesn't seem to work in meteor

help
1

I’m trying to do graceful shutdowns as my application receives SIGINT or SIGTERM. I basically do this:

    setupSignalHandlers = () => {
        process.on("SIGTERM", () => {
            console.log("SIGTERM received");
            this.stop();
        });
        process.on("SIGINT", () => {
            console.log("SIGINT received");
            this.stop();
        });
    };

And have my cleanup code in this.stop(). If I start this using node myFile.js, it works just fine. If I run it in a meteor process, my callbacks are not triggered by the signals. I can’t find any examples of a meteor specific way of accomplishing this either. Does anyone have any similar experiences, or have successfully accomplished something to the same effect as what I’m trying?

1 Like
2

Did you try wrapping the callback with Meteor.bindEnvironment()

3

As in

...
process.on("SIGTERM", Meteor.bindEnvironment(() => { /* intercept logic */ }));

?

I tried now with no result. But also wasn’t aware of the bindEnvironment method, so I might not have set it up correctly.

4

I think this might be of interest here:

5

I’ll give it a go, but looks as if that lib is also relying on process.on(signal) to function.

6

The package might also be in need of an update @grubba @fredmaiaarantes

2 Likes
7

I’ve used the ddp-graceful-shutdown package in many projects before without any issues. Its code is pretty small, so @jlugner might want to take a look at its repository.

I just installed it in a simple project to test it on Galaxy, and it worked!

I’m using the code below that I added in a new file named shutdown-handler.js. The purpose of the package is to shut the DDP connections down gradually, so I’m overriding installSIGTERMHandler() so I can do something before they are closed.

import { Meteor } from 'meteor/meteor';
import { DDPGracefulShutdown } from '@meteorjs/ddp-graceful-shutdown';

class SimpleDDPGracefulShutdown extends DDPGracefulShutdown {
  installSIGTERMHandler() {
    process.on(
      'SIGTERM',
      Meteor.bindEnvironment(() => {
        const gracePeriod =
          process.env.METEOR_SIGTERM_GRACE_PERIOD_SECONDS || 30;

        console.log(
          `Received SIGTERM. Shutting down in ${gracePeriod} seconds.`
        );

        this.closeConnections({ log: true });
      })
    );
  }
}

new SimpleDDPGracefulShutdown({
  gracePeriodMillis: 1000 * process.env.METEOR_SIGTERM_GRACE_PERIOD_SECONDS,
  server: Meteor.server,
}).installSIGTERMHandler();

I’m importing it in main.js, which is my server entry point. And that’s it.

import { Meteor } from 'meteor/meteor';
import { Migrations } from 'meteor/percolate:migrations';

import './shutdown-handler'; //Here
import './db/migrations';
import './tasks/tasks.publications';
import './tasks/tasks.methods';

/**
 * This is the server-side entry point
 */
Meteor.startup(() => {
  Migrations.migrateTo('latest');
});

When my container is being stopped, I can see the message below on Galaxy logs:

Screenshot 2023-11-14 at 17.40.59
Screenshot 2023-11-14 at 17.40.591792×378 98.8 KB

My current grace period is 40 seconds:

Screenshot 2023-11-14 at 17.47.45
Screenshot 2023-11-14 at 17.47.452038×496 27.4 KB

1 Like
8

I can’t seem to replicate… Every now and then I reach the callbacks (~10% of the time?), but even in those cases it immediately exits. I’m trying to use it in my background worker. Idea is to let it finish any running jobs when a signal is received, but not start any new. This is virtually what it’s doing, but with sleeps in place of any actual job logic:

import { DDPGracefulShutdown } from "@meteorjs/ddp-graceful-shutdown";
const SLEEP_TIME = 100;

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

export class Worker extends DDPGracefulShutdown {
    running = false;
    grabFirstQueuedJob = async () => {
        console.log("Grabbing first queued job");
        await sleep(1000);
        console.log("Pretending to grab first and running it");
    };

    executeJob = async () => {
        console.log("Executing job");
        await sleep(10000);
        console.log("Done executing job");
    };

    run = async () => {
        if (!this.running) {
            console.log("Worker stopped running");
            return;
        }

        try {
            await this.poll();
            setTimeout(this.run, SLEEP_TIME);
        } catch (e) {
            console.error("An error occurred while polling:", e);
            setTimeout(this.run, SLEEP_TIME);
        }
    };

    stop = () => {
        console.log("Stopping worker");
        this.running = false;
    };

    setupSignalHandlers = () => {
        process.on(
            "SIGTERM",
            Meteor.bindEnvironment(() => {
                console.log("SIGTERM received");
                this.stop();
                this.closeConnections({ log: true });
            }),
        );
        process.on(
            "SIGINT",
            Meteor.bindEnvironment(() => {
                console.log("SIGINT received");
                this.stop();
                this.closeConnections({ log: true });
            }),
        );
    };

    poll = async () => {
        await this.grabFirstQueuedJob();
        await this.executeJob();
    };

    start = async () => {
        console.log("Starting worker");
        this.setupSignalHandlers();
        this.running = true;

        await this.run();
    };
}

if (process.env.IS_WORKER) {
    const worker = new Worker({
        gracePeriodMillis: 1000 * 30,
        server: Meteor.server,
    });
    worker.start();
}

The file is loaded first thing in my startup script. I’m running everything locally

9

How are you sending SIGTERM to the process locally? Would you be able to deploy and test it on a server?

10

Yeah, locally. Sending signals using ctrl + c. Works for regular node without issues. I’ll have a go at doing it on a server (Heroku).

11

Same result I’m afraid. Callback doesn’t get called, executing stops immediately.