How do you handle job's code?

deployment
1

Hello,

I have an app which runs on galaxy.

I am hitting a problem where I need from time to time to write jobs to change datas on my MongoDB.
Currently I need to redeploy on galaxy after testing the code locally.

I am looking for a way to have a separate instances where I can simply deploy my meteor app without annoying my users so that I can have my new jobs run immediately.

I am using wildhart/meteor.jobs ro run jobs so maybe I can set the serverId of the queue to be the one of the server I will be deploying.

I am pretty sure that everyone have this problem ( writing codes for chore purposes ) but I do not find any good solutions

2

My replies here might help give you an idea: Reactivity problems after migrating to meteor 3.0.3 - #13 by rjdavid

Our jobs are completely separate from the main user-facing app

3

Nosqlbooster has some scripting ability - you can basically write js code which manipulates your data without the need for deployment overhead. You can develop your script against a copy of your database

4

We also started doing something similar to @rjdavid recently. We had tried various other queues in the past but Bull is the one that feels really well thought out and is in active development. Bullboard is also nice for keeping tabs on them.
We kept a single app and then assign roles like webapp or jobs to different instances. Ideally we’d have separate apps which could be launched in parallel with rollbacks if any of them fail - but we run on elastic beanstalk and the options for orchestration are limited.
With the updates to Monti planned to automatically instrument job queues this should become even nicer.

5

Set up a separate staging environment on Galaxy could help - you can test job changes there before deploying to production. The wildhart/meteor.jobs package should let you use a different server ID for the staging queue

6

The wildhart/meteor.jobs package should let you use a different server ID

@metrich how can you setup that ?

I mean if I have a job.company.com app on galaxy how can I use setServerId to be sure that only jobs are run on this server ?

7

I use webapp to handle jobs, then I setup cronjob on a private machine which calls the api. By using this setup, I don’t have to deal with “serverId” issue.

8

Hello @minhna which API are you talking about ?

9

“API” I mean webapp I use. For example

webapp.connect('"jobs/send-messages", async () => {
  // ...
});

to run it, just make a GET/POST request to that url. You might want to setup some security checks to make sure not everyone can call it.

10

Ok and then you just find all jobs that are pending and run them inside ?

11

We have split our app into 2 parts, one is the client facing app, and the second is an admin app. There are 2 meteor instances which share the same DB. This gives us the ability to modify and deploy the admin app independently of the client facing app. This could be what you need?

12

Yes, but not that easy to do because it required to at least have collections in private package to be consumed inside both app.
I am trying to find a way to do it with same codebase and to tell GitHub - wildhart/meteor.jobs: A simple job scheduler for Meteor.js to consume jobs only on a certain server based on env variable for example or meteor settings.

13

We have a group of apps that use the same collections using git-subrepo

14

I use an updated / modified version of this: GitHub - konecty/meteor-multiple-instances-status: Keep a collection with active servers/instances
I allocate instances names using a slug generator that looks into the Instances DB and calculate unique names/slugs.
Something like this:

Meteor.startup(async () => {
  const slug = await createSlug('act', 'instance')
  InstanceStatus.registerInstance(slug)
    .catch(err => console.log(err))
})

You can search for an Instance by name, get its Id and compare with your running instance id.

const targetInstance = (await Instances.findOneAsync({ slug: whoShouldRunTheJobs }))?._id
const thisInstanceId = InstanceStatus.id()
if (targetInstance === thisInstanceId) {
  // go ahead, run the jobs.
}

You can create a logic in your instance name generation for instance with sequence numbers such as server1, server2, server3 etc and run specific jobs with specific names with the same codebase.