Clearing server-side timeouts on live reload

help
1

Hi all!

I have a simple piece of server-side code like this:

Meteor.startup(() => {
 testData();
});

const testData = async () => {
  console.log('Processing data');
  // do stuff which usually takes around 3 to 5 minutes, depending on remote server responses
  Meteor.setTimeout(testData, 30 * 1000);
};

The intention here is to run my testData() function on a regular interval, but only after the function processes all the data, which takes unknown amount of time, so triggering it with a cron job doesn’t make much sense in my use case.

When I run my Meteor app for the first time, I can see “Processing data” in the logs every 30 seconds. However, when I update my code and save it, I see in the logs that app reloads and “Processing data” log line doubles in frequency, I guess because the timeout is never actually cleared so I see both log lines from previous run and after live reload. And this happens every time I update my code and save it, resulting in unwanted timeouts adding up and constantly triggering my function more and more often. I’m aware that this most likely won’t be the case in production app, but I’d like to avoid this behaviour during the development if possible.

My questions here are:

  • is there some kind of live reload hook which I could use to clear all timeouts or is there some other recommended way to tackle the issue I’m facing in development environment?
  • is there maybe a completely different approach to achieve functionality I need (instead of using Meteor.setTimeout)? I took a look at quave:synced-cron and msavin:sjobs packages but the former didn’t implement setTimeout functionality and the latter can’t be installed with Meteor 3.

Any suggestion, idea, recommendation, help is extremely appreciated!

Thanks!

2 Likes
2
  1. You can test this on your server side. Whenever you save changes in your server code, this will be triggered.
const com = () => console.log('restarted')
process.on('SIGTERM', com)
  1. You can loop over a number of records and re-loop once you reach the last record.
2 Likes
3

Thanks for pointing out catching SIGTERM, that didn’t occur to me :slight_smile:

I’m not sure if I completely understood your second point, but if I did, I think that’s maybe not applicable in my use case.

After some experimenting, I ended up with something like this for now:

let timeouts = [];  // Array to hold all the timeout references

process.on('SIGTERM', () => clearTimeouts());

Meteor.startup(() => {
 testData();
 testOtherData();
});

const clearTimeouts = (name = null) => {
  if (name) {
    // Clear only timeouts for given function
    timeouts.map(i => i.name === name && Meteor.clearTimeout(i.timeout));

    // Remove all timeout references for given function name from
    // 'timeouts' array to avoid array growing in size
    timeouts = timeouts.filter(i => i.name !== name);
  } else {
    // Clear all timeouts from 'timeouts' array and reset it
    timeouts.map(i => Meteor.clearTimeout(i.timeout));
    jobs = [];
  }
};

const testData = async () => {
  console.log('Processing data');
  const name = 'testData';
  const timeoutDelay = 30;

  // do stuff

  // Clear all timeouts for this function to avoid 'timeouts' array growing in size 
  clearTimeouts(name);

  // Set new timeout and add its reference to 'timeouts' array
  const timeout = Meteor.setTimeout(testData, timeoutDelay * 1000);
  timeouts.push({ name, timeout });
};

const testOtherData = async () => {
  console.log('Processing other data');
  const name = 'testOtherData';
  const timeoutDelay = 60;

  // do stuff

  // Clear all timeouts for this function to avoid 'timeouts' array growing in size 
  clearTimeouts(name);

  // Set new timeout and add its reference to 'timeouts' array
  const timeout = Meteor.setTimeout(testOtherData, timeoutDelay * 1000);
  timeouts.push({ name, timeout });
};

I still think this can be done in a more elegant way, but it looks to me like it will do the job in my simple scenario. If someone has an additional suggestion, I’m very open to hear it!

Thanks again!

4

Your problem is not just reloads during development. In production, once you have multiple instances of your server running, each instance will run your loop/timer.

There are a lot of npm-based jobs/tasks queue that you can use if you cannot find anything from atmosphere that suits your needs

1 Like
5

Oh, thanks a lot for pointing this out! I would probably find this out in a hard way otherwise :sweat_smile:

I actually noticed the other day that there are some forks of msavin:jobs (I used that one long time ago so I’m kinda familiar with it) trying to update it for Meteor 3, so I went ahead with one of them (StorytellerCZ version) and gave it a try. Managed to install it as a local package and refactored my code from previous snippet to this:

process.on('SIGTERM', () => console.warn('SIGTERM'));

Meteor.startup(async () => {
  Jobs.register({ longRunningFunction });

  Jobs.run('longRunningFunction', { singular: true, unique: true });
});

const longRunningFunction = async function () {
  // do long running stuff
  this.reschedule({ in: { seconds: 30 } });
}

I’ve noticed that I actually still have the same problem with live reloads in dev env, but at this point I’m really puzzled why is that. Somehow I thought that jobs package will handle this scenario in a better way. And now you mentioned production instances, I’m not convinced that everything will work perfectly fine there either. Maybe I can just clear all my jobs during SIGTERM, same as I was clearing timeouts earlier, but not sure if that’s a smart thing to do?

Additionally, I see that reschedule() doesn’t really work as I thought it will – if the function in my example is running longer than 30 seconds, the job will start immediately after my function ends instead of waiting for 30 seconds. I can solve this by passing explicit date to reschedule() and in the end this is a completely different topic.

I’ll definitely try to take a look at available npm packages as you suggested, I was just hoping to use something developed for Meteor…

Thanks again, back to experimenting now :slight_smile:

EDIT:
No, clearing all the jobs during SIGTERM didn’t actually help, I still see previous jobs continuing after live reload as well as new one starting :confused:

6

I recommend you check out quave:synced-cron. The synced-cron package is simple and has been in use in Meteor apps for ages. The quave version has it updated to Meteor 3.

1 Like
7

Hey, thanks for suggestion, I did take a look at that one, but what I need is to trigger my function after n seconds after it ends running. I was hoping to avoid calculating specific dates when to next run my “jobs” and was thinking that msavin:jobs reschedule({ in: { seconds: 30 } }) would do the trick, but it looks like I was wrong about that. The other thing is that I used that library long time ago so I’m familiar with it and I like the job management interface a lot.

But given that I would have calculate exact time for the next run anyways, I might give synced-cron a shot.

Can you tell out of your head, would this library help me with my original issue from my first post – stopping previous (long-running) jobs during live reload in development?

Thanks again!

EDIT:
It looks like synced-cron didn’t help me with my initial live reload issue either :frowning:

8

So, back to my original issue with observing long running jobs continuing after live reload and same ones starting over, resulting in running the same jobs/functions multiple times…

I started that project by bootstrapping Vue app:

$ meteor create --vue my-app

To convince myself that I’m not absolutely crazy, I just tried to create a completely fresh minimal Meteor project:

$ meteor create --minimal my-app-test

after which I copied my existing app code to it and tried to add one by one Meteor and NPM packages and observe the behaviour.

To my shock and surprise, the issue I originally had never happened until I added a specific Meteor package – jorgenvatle:vite-bundler.

I tried to do the same with multiple “minimal” Meteor projects and NPM packages combinations (since Vue bootstrap project has quite a few of them outdated) and I ended up with literally all Meteor and NPM packages installed from Vue bootstrap version in a minimal version except jorgenvatle:vite-bundler and never saw the issue with live reload. The moment I added jorgenvatle:vite-bundler the issue happened again.

Tagging @jorgenvatle here, please read my original post and let me know if you have a suggestion how to solve this issue. I can provide exact steps to reproduce it if needed. Let me know if you’ll need any more info from me! Hjelp meg venn, takk! :slight_smile:

9

Is there any reason why you cannot use a promise so that you can await your data processing and execute as soon as the data processing has veen completed?

10

I am actually using promise (awaiting in my function until all data is fetched) and the purpose for my initial timeout was to avoid hitting rate limit on remote server, nothing more than that. But my problem was never a timeout or job queuing, it was that my job (function) was continuing to run after live reload instead of stopping, so after live reload I would see the old one running and the newly triggered one. After multiple live reloads, this would multiply, of course.

11

To reproduce the issue I’m facing, perform following simple steps:

  1. Create new Meteor project, e.g.
$ meteor create repro --minimal --release 3.0.1
$ cd repro
  1. Replace the content of server/main.js with following code:
import { Meteor } from 'meteor/meteor';

Meteor.startup(() => { test() });

const test = async () => {
  console.log('Processing data');
  Meteor.setTimeout(test, 2 * 1000);
};
  1. Run your app:
$ meteor

  1. Change the "Processing data" string in your console.log() line, save it, and observe in logs that after live reload you will see only new string printing out. Change the string multiple times and observe that after saving the file, you’ll always see only the latest string printing out in logs.

  2. Stop your Meteor app.

  3. Add jorgenvatle:vite-bundler package and its dependencies:

$ meteor npm install vite@4 meteor-vite -D
$ meteor add jorgenvatle:vite-bundler
  1. Create a minimal vite.config.js file in the root of your project (optional, just to avoid Vite’s warning printing out):
import { defineConfig } from 'vite';

export default defineConfig({
  plugins: [],
  meteor: {
    clientEntry: 'client/main.js',
  },
});
  1. Run your app:
$ meteor
  1. Do the same thing as in step 4 above, but this time observe that after changing the string in console.log() and saving the file, you will see the old string and the new string printing out in logs. Change the string multiple times and save your file after each change and observe that after multiple live reloads you’ll see the string from each and every change still printing out in logs.

@jorgenvatle Added this simple reproduction steps, could have done it yesterday but it was very late here and I was pretty much exhausted. Please let me know if there is a solution for this already or if you think this might be really some unwanted behaviour coming from jorgenvatle:vite-bundler package. Takk!

1 Like