Running Multiple service in a single meteor app bundle

help
1

I am new to meteor and would like to know how do I run multiple service or multiple process in parallel.

Basically I have a method that takes 5min to complete the process. Its a CPU intensive process and due to this my meteor app is getting blocked for going for any other process.
So want to run it in a parallel service so that it will not getting blocked while executing the task.

Flow I want to achieve:

  1. Request from the front end to execute a task.
  2. It will be received in backend and add to the task queue.
    2.1 Once the task execution is complete it will update the result in db and the same will reflect in front end.
    2.2. Check for next task.
  3. On front end side it will show processing for that particular task and user should able to create more task as per his wish.

I basic idea of fibre, child_process, fork etc but couldnot able to apply the same here. As well as not getting proper guide for the same. Kindly suggest me the process, any tutorial or any book will be great. Thanks in advance.

2

hope this helps: https://nodejs.org/docs/latest-v12.x/api/worker_threads.html

3

You could also solve it by moving/migrating your method to AWS lambda, which I think should be the preferred method.

While you can indeed use Node.js worker threads, that would constitute a “scale up”, but what you might need is a “scale out”. Consider that case where multiple users want to execute that particular CPU intensive process, where the parallelity may easily exceed the number of threads allotted to your container.

AWS lambda, however, could give you an infinite autoscaling just with respect to these tasks, without having to autoscale your Meteor instances just because of those.

4

Tasks like this must be moved on a job queue.

1 Like
5

Lambda would be the cheapest and most capable for this type of tasks. You can create a Lambda Interface using ClaudiaJS(https://claudiajs.com/).
You can use Env vars in Lambda in order to connect to your DB and run the queries (with a Node driver).
From your Meteor server you do a POST into your Lambda interface with the parameters required and subscribe to … an ‘updated’ field in your DB or any other way to know the operation was completed.

If you get lost in ClaudiaJS or Lambda just let us know and I’ll help you with getting you set up.

2 Likes
6

Just announced: Go multi-core with nschwarz:cluster