Meteor method queue processing

help
1

I’m working on the functionality, which have following feature -

  • Method ‘generateReportPdfs’ which generates multiple PDFs based on the report collection.
  • After generating the pdf. I’m storing the pdf files in AWS s3 bucket.
  • This method returns all the uploaded pdf files URLs.

Currently, I’m using promise.all() method in method ‘generateReportPdfs’ so all the PDF generation process running in parallelly and the promise return the response when all pdf url generation process successfully finished.

Parallel processing is best when we generate the 5-6 PDF on the server side. But, it’s not the best solution, If we need to generate 98 PDFs. Because It makes server busy.

Can we modify the method to generate the pdf queue basis? The method ‘generateReportPdfs’ returns the pdf URL when one job has finished and 2nd pdf when 2nd pdf job is finished. this process continues until 98th pdf finished.

I really appreciate If someone has better idea to implement queue processing.

1 Like
2

I think it’s possible by using async/await.

1 Like
3

If you’re using Promise.all, then you must be passing in an array of Promises. In which case, you can just iterate over the array and use async/await as @minhna suggests :

arrayOfPromises.forEach(async (prom) => {
  await prom;
});

Note: if you want the results from each Promise, you’ll need to modify that code to capture those in a results array.

4

@robfallows

But how meteor method returns multiple time? If one promise resolved then need to return the generated link to the frontend. How can we do this?

5

So, Promise.all returns an array of results, with each element of the results array corresponding to the initial element in the array of Promises:

const results = await Promise.all([ promise1, promise, promise3 ]);
// results[0] = result from promise1
// results[1] = result from promise2
// results[2] = result from promise3

I guess, you’re returning results from your method?

If you refactor to:

Meteor.methods({
  async myMethod() {
    const results = [];
    [ promise1, promise, promise3 ].forEach(async (prom, i) => {
      results[i] = await prom;
    });
    return results;
  },
});

you’ll get exactly the same results returned - it will just take longer, because the Promises are processed serially.

6

@robfallows That method returns once which include the result. (result is an array that contains the s3 links). But I want the approach that returns the result to client when one promise finished let’s say promise 1. and returns again when the second promise finished.

7

Technically, you could do that in a Meteor.method with a Generator, but you’d need to repeat the Meteor.call multiple times to get each value in turn. I’ve done this myself, so I know it works - but quite honestly, it’s going to make debugging harder.

I see I already answered you here:

8

I think you can call method multiple times from client.
Call method 1, wait until it finish, call method 2…

9

@minha @robfallows I’m doing the same as you suggested the calling method multiple times. I just want to Is there any other way to implement this?

10

here is a example:
the method run on server:

Meteor.methods({
  'test.task': async function({index}){
    console.log('do task with index: '+index);

    function waitXSeconds(x) {
      return new Promise(resolve => {
        setTimeout(() => {
          resolve(x);
        }, x * 1000);
      });
    }

    await waitXSeconds(2);
    return index;
  }
});

and the client call the method:

onClick(){
    const callWithPromise = (method, params) => {
      return new Promise((resolve, reject) => {
        Meteor.call(method, params, (error, result) => {
          if (error) reject(error);
          resolve(result);
        });
      });
    }

    const callMethod = async ()=>{
      const result1 = await callWithPromise('test.task', {index: 1});
      console.log(result1);
      const result2 = await callWithPromise('test.task', {index: 2});
      console.log(result2);
    }

    callMethod();
  }
11

I’m trying to do something slightly different and have been scouring the forums and await/async examples trying to figure it out. I’m trying to make a “self-blocking” Meteor Method that blocks future calls to itself until earlier calls from the same client have resolved

I have a Meteor Method that can get called repeatedly by the same client. The method does some database operations and needs to await those database updates before it gets called again. According to the docs, Meteor Methods are supposed to work this way (self-block unless Meteor.unblock() is called), but they always all get called right away.

Taking @minhna’s example - I have something like:

Meteor.methods({

   selfBlockingMethod: async function() {

        console.log("selfBlockingMethod: I was called...");

        //  This would really be a bunch of database updates that needs to happen
        //  before the method is fired again
        function waitXSeconds(x) {
            return new Promise(function(resolve, reject) {
                setTimeout(function() {
                    resolve(x);
                }, x * 1000);
            });
        }

        await waitXSeconds(3);

        console.log("selfBlockingMethod: I am finishing.");
    }
});

So then on the client if I fire off three calls in a row immediately like this:

Meteor.call("selfBlockingMethod");
Meteor.call("selfBlockingMethod");
Meteor.call("selfBlockingMethod");

The output I get is:

selfBlockingMethod: I was called…
selfBlockingMethod: I was called…
selfBlockingMethod: I was called…
selfBlockingMethod: I am finishing.
selfBlockingMethod: I am finishing.
selfBlockingMethod: I am finishing.

The initial “I was called” statements are all shown right away. I’m expecting the output to be:

selfBlockingMethod: I was called…
selfBlockingMethod: I am finishing. (three seconds later)
selfBlockingMethod: I was called…
selfBlockingMethod: I am finishing. (three seconds later)
selfBlockingMethod: I was called…
selfBlockingMethod: I am finishing. (three seconds later)

Can this use-case be achieved? I would think having the async on the actual Meteor Method with the await below would do this.

I have javascript to disable buttons and all of that, but I’m trying to write my code to prevent hacks on the console as well.

12

To prevent that kind of hack, you can use DDPRateLimiter package.

13

Actually though I may want the user to have the ability to call this function quickly and repeatedly. It’s just needs to block for however many milliseconds it takes to update the DB.

Calling the three console lines all at the same time is a test that my code needs to pass. In reality, I disable a UI button until the Meteor Method returns. But if someone wanted to use the console (and/or just as a test) it needs to still work with integrity. So I need to still solve this problem.

@robfallows Any ideas?

14

There is another solution. Inside the method, you check for user (logged in user), then if user call the method, at the beginning, you update the collection in the database to mark that user is calling the method. At the end of the method, you remove that record or mark it finished. So in the method, you check if the user are marking another method call, then you can decide what to do.

15

That’s exactly how it happens if you use async methods. I documented this behaviour here:

16

@robfallows Thanks for the article link. I had read that, but reading it again helped. I think I’ve narrowed down the issue.

Here’s the problem: I have a Meteor Method that calls a server function that does an asynchronous database update. So if I have absolutely no async/await code, repeated calls to the Meteor Method would naturally block each other except for the fact that I am doing an asynchronous database update that returns immediately. So the Meteor Method doesn’t block the database update. I need to block each Meteor Method call until that database update resolves. So what I did was wrap the database update in a Promise - which works (I’ve swapped out the database update code with a setTimeout which illustrates the same issue):

Meteor.methods({
   selfBlockingMethod: async function(order) {
      console.log("selfBlockingMethod:", order, "was called...");
      //  Call server function that runs a Promise-wrapped asynchronous function
      await _promiseWrappedSetTimeout();
      console.log("selfBlockingMethod:", order, "has finished...");
   }
});

var _promiseWrappedSetTimeout = function() {
   return new Promise(function(resolve, reject) {
        Meteor.setTimeout(function() {
            console.log("_promiseWrappedSetTimeout: timeout is up!...");
            resolve(true);
        }, 3000);
    });
}

On the client console, if I run Meteor.call("selfBlockingMethod", 1), I get the following expected output on the server:

selfBlockingMethod: 1 was called...
_promiseWrappedSetTimeout: timeout is up! (three seconds later)
selfBlockingMethod: 1 has finished...

Seems like it worked right? It does for one call. The problem is if I fire this off in the console:

Meteor.call("selfBlockingMethod", 1);
Meteor.call("selfBlockingMethod", 2);
Meteor.call("selfBlockingMethod", 3);

Because the Meteor Method is set to async, I get this which is not what I want:

selfBlockingMethod: 1 was called...
selfBlockingMethod: 2 was called...
selfBlockingMethod: 3 was called...
_promiseWrappedSetTimeout: timeout is up! (same time 3 secs later)
_promiseWrappedSetTimeout: timeout is up! (same time 3 secs later)
_promiseWrappedSetTimeout: timeout is up! (same time 3 secs later)
selfBlockingMethod: 1 has finished...
selfBlockingMethod: 2 has finished...
selfBlockingMethod: 3 has finished...

I need the Meteor Method to keep self-blocking like a normal Meteor Method. So I tried removing the async off the Meteor Method and putting it on a wrapper function defined within the method:

Meteor.methods({
   selfBlockingMethod: function(order) {
      console.log("selfBlockingMethod:", order, "was called...");
      
      //  Create a wrapper function that's async
      let _wrapperFunction = async function() {
         if(Meteor.isServer)
            //  Call server function that runs a Promise-wrapped asynchronous function
            await _promiseWrappedSetTimeout();
         }
      }

      //  Call the wrapper function
      _wrapperFunction();

      console.log("selfBlockingMethod:", order, "has finished...");
   }
});

However, this doesn’t work. The Meteor Method finishes instantly and doesn’t await the setTimeout to complete like before.

selfBlockingMethod: 1 was called...
selfBlockingMethod: 1 has finished...
_promiseWrappedSetTimeout: timeout is up! (three seconds later)

How can this be achieved!? I feel like I’ve tried everything. I’m probably not doing something right.

17

You could also use something like Steve Jobs to run the task, and perhaps pub/sub or something else to notify the end user when its ready.

18

I’m not at my computer right now and can’t test this myself, but you could try using a standard method (not async). Instead of using

const x = await somePromise

Do

const x = Promise.await(somePromise)

Don’t forget to

import { Promise } from 'meteor/promise'
19

@shubhama I came across this problem in the context of providing status updates of long running methods on the server, I ended up implementing it as a package, you could probably use a similar technique (possibly even the same package) to publish results to the client as they are ready from a single call. Internally it publishes data to the client and your method callback gets triggered multiple times

https://atmospherejs.com/znewsham/observable-call

20

Depending on how quickly this needs to be, the guaranteed way of doing this is to make the next method call only when the previous call has returned a result. You can then use an async method and it will work.

That’s easier if you Promisify the call and use async/await, rather than get into callback hell.