Hypothetical long running Fibers question

performance
5

Exactly - I know you can yield a fiber (though I don’t know if you can do this from outside the running fiber, which would kill this idea outright), but I’m thinking of an interface like this

LimitedTaskRunner.run(fn, cpuThresholdLimit)

or even

LimitedTaskRunner.run(fn, dontStartIfCpuIsAbove, pauseIfCpuIsAbove)

So we don’t even start a job if the CPU is highish (e.g., 50%) but if we do start it, we can stop it if things get too heavy (e.g., 70%). This might stop us from thrashing the CPU when we’re already close to our limits. One could even add some machine learning to something like this, to determine which functions (with which arguments) are likely to be heavy so you can estimate how much this will impact current CPU usage.

1 Like
6

Seems like you can’t yield from outside the fiber, so we’d probably have to implement a function like Fiber.maybeYield which can be cauesd at points that the function can be paused at, and it checks the CPU usage and either yields or continues. So this wouldn’t be fully generic, but I think any method that is heavy would have potential yield points, either in a loop or in the rest of the code. But it would make it more difficult to reuse existing functionality

7

Yeah, well I admittedly don’t have much experience in that area, but I find the idea interesting, so just sharing thoughts.

It sounds like what is being asked here is a generic framework that manages jobs execution based on available resources (CPU, idle time patterns, among others) at run-time. That framework make sense on tight resource constraints per VM where you’ve jobs that can yield. I’m just wondering if the added complexity worth the benefit. Is there a reason why not putting those jobs on separate VMs?

8

Hmm, I’m not sure how much complexity it would cause (after the initial package implementation). For example a function like this:

function myLongFunction() {
   MyCollection.find().forEach((item) => {
        //do something expensive
        LimitedTaskRunner.maybeYield();
   });
}

Meteor.methods({
    myLongFunction() {
        LimitedTaskRunner.run(myLongFunction, 60);
   }
});

I’m not overly familiar with custom VMs, but how easy is it to spinup a new VM and wait for the result? I guess you could have an entirely separate Meteor server, and put your common code in a package, and the purpose of the secondary meteor server is to run these long running functions - it can run at 100% without impacting regular performance, but wouldn’t be able to limit a functions execution based on the current user, for example, the package could expose an API like this:

LimitedTaskRunner.yieldIf((userId, currentCpuUsage) => {

});

This way you can easily define custom logic as to whether or not to yield on a per user basis. This on its own is useful (e.g., limit free users, but dont limit payed users, or limit free users and limit payed users less, or check to see how many functions a specific user is currently running and yield or not). What would be even more useful is if there was a way of detecting how much CPU time a user had used, given that Kadira is able to compute some of these values, I think we could also sum up the total CPU time and yield only if a user isn’t being excessive.

A tiny bit more detail on my thought process: The project I’m working on up until now is always used by companies, who pay quite a reasonably large amount, and there is a limited number of them, and they are online typically at different times of the day/month/year in some cases. But we just opened up the system to turnkey users (e.g., put in a credit card and go) and I worry that individual users could end up consuming a lot of resources (or a lot relative to the amount they pay).

I am completely open to other suggestions too

1 Like
9

Yes I personally would do it that way without cluttering the app logic with performance management. Create a micro-service for those jobs, you can use Meteor (something like the Steve Job package) or pure node, and you can utilize all the CPUs and create a queue for the jobs. That way you can monitor this server on it’s own, scale it vertically etc.

10

I can see how it would work well on a meteor server, just a remote method call and wait for the response, but with a node process I would think the extra code necessary to spinup new processes (and potentially new servers) would be quite complicated, and potentially expensive if your primary server was currently idle anyway), and wouldn’t handle the case of per-user (or possibly per connection) resources.

If the only concern is CPU and memory (which right now it is) something like AWS lambda would work well, though all code which currently is Meteor dependent would need to be converted to pure node (not super hard, but not trivial either). This still wouldn’t give you per user/connection rules, but would be cheaper and probably easier than spinning up a new server.

We currently use both lambda and spot servers to handle very specific workloads that are REALLY heavy and CPU bound, but I think in most cases it would be quite expensive.

The other consideration is that CPU usage isn’t the only metric of interest here (whether per user, per connection, per DDP call or globally). Another useful metric would be DB calls - this is somewhere that the DDP rate limiter doesn’t really help, a method call that triggers a single DB update, or returns a single DB document is treated the same as a call which updates 1000 documents, or returns 1000 documents. We have some methods that allow “bulk actions” to be taken over a range of documents, rate limiting doesnt help here.

I think I might have a crack at implementing this as a package. I don’t think it would be particularly hard just for CPU usage initially.

1 Like
11

Don’t overcomplicate things.

You can create a document that represents the progress of your long running task. Start the task in a Meteor.defer(). Update this document from time to time, which will also yield. Subscribe to changes in this document. Add a createAt and modifiedAt field, and decide, internally to your application, how long to wait until the document is last modified before deciding that the job is dead (i.e., “fail-safe” for the long running process).

The job will die if you take down the node, which is true for any architecture. Use the job document to recover what was started, and maybe restart it somehow. Use it for storing progress. Don’t use other packages, this stuff is incredibly straightforward.

12

Thanks for the suggestion, but this won’t cover the ability to limit different users differently, nor the ability to scale throttle based on current usage

13

Think about what you’re saying. Throttling usage on a web server? You should try to run at 100% CPU and RAM, otherwise you’re wasting your precious AWS dollars :slight_smile:

On a more technical note. Don’t try to write a scheduler. Don’t try to schedule fibers or servers or processes. Throttle based on job size, which you can determine before running the job—that is, don’t start big jobs for users who don’t pay. Or set a time limit. Your load will be proportional to the number of jobs in flight, which will be proportional to average lifetime of the jobs, which will be proportional to the size of the job.

If this is a numeric task, or something that resembles a tree search, it’s beneficial to use a separate process if reducing your transient memory usage is absolutely essential. Alternatively, if you want to run it in the same process as the web app, remember that node (like java) stores everything on the heap: you have to use object pooling to keep transient memory low. However, I’d warn that there are a lot of pitfalls to doing so on node.

14

I would love to run at 100%, but I don’t want other users to be stuck waiting for someone elses job to finish, that is non time sensitive. I invite you to try this simple test:

Meteor.methods({
longFunction() {
    this.unblock();
    const future = new Future();
    Meteor.setTimeout(() => {
      let res = 0;
      for (let i = 0; i <= 1000000000; i++) {
        res += Math.random() * Math.random();
      }
      future.return();
    }, 5);
    return future.wait();
});

Open a tab, call that method - and try opening another tab - your tab wont load until 15-30 seconds later, when this method finishes. This is of course an exagerated example, but it ilustrates things nicely. Some methods can be delayed to ensure responsiveness of the app as a whole.

Now, this isn’t to say this is the best approach in general, its likely that spinning up an entirely separate process, or server will be better - but I’m interested to see if this is possible, and what the drawbacks are.

15

Yes, do this:

Meteor.methods({
longFunction() {
    let jobId = Jobs.insert({progress: 0});
    Meteor.defer(() => {
      let res = 0;
      for (let i = 0; i <= 1000000000; i++) {
        res += Math.random() * Math.random();
        if (i % 1000 == 0) {
            // This will yield in the way you need it to.
            Jobs.update(jobId, {$set: {progress: i}});
        }
      }
    });
    return jobId;
});
16

It will yield, and immediately resume - that isn’t what I want it to do. I only want it to yield if it needs to, and I only want it to resume if it the server has capacity to

17

What do you mean by immediately resume?

It’s not going to keep processing until the database has updated. During that time, it may choose to process any number of other waiting tasks on the event loop.

Fiber_Id         Progress         Unparked_By        Parked_By
0                    1000         Database return    Database call
1                       ~         Method call        Method return to DDP
0                    2000         Database return    Database call
2                       ~         Method call        Method return to DDP
0                    3000         Database return    Database call

What do you think the operating system scheduler does?

What does it mean, yield if it needs to? Just think about it. Supposing these were different processes, how do you think the OS knows it “needs to” yield?

Real processes/threads are parked (yielded) all the time!

Resume if the server “has capacity”? I know you want just a simple answer here, but again, by what meaning?

I think if you’re not sure what the answers to these questions are, by all means, start a node process. Operating system scheduling works for everyone, and it’ll work for you. If you want to park and unpark fibers, by all means, do that too. If you want to have 100% control of when a thread/fiber (called strands in the literature) run and don’t run, you need a thisStrand.parkAndUnpark(otherStrandToResume) primitive. Most applications use queues as a high level object to achieve this.

The most important thing to realize is that if you run the code I gave you, everything’s going to be fine and dandy. Under the hood, calls to a Mongo.collection park the caller until the database is done writing. Between the database update and its return value, the event loop will keep getting processed. If you want to like, idle more, just call Fiber.sleep. If you want to “yield more often,” reduce the modulo operand (the part after the %), but that’s pretty wasteful. I don’t see why you’d want to do that, since if it could do more work, it will!

18

I’m not really looking for an answer - unless the answer is “here is a library that already does it” or “here is someone who tried it and all the reasons it cannot work”.

The operating system scheduler is irelevant here - unless I’m totally misunderstanding how node works, everything runs in a single process. Not just one process but a single thread - so given that pretty much the only thing running on my server is Meteor, the operating system scheduler isn’t useful here. Feel free to correct me on this - it would be a massive weight off my shoulders, but I’m about 99% sure I’m correct.

“Yield if it needs to” means exactly that - yield only if some condition is met - if you look at your example, the condition is trivial, yield every 1000 jobs, and resume as soon as the DB update is complete (a couple of ms later? Less?) - let’s ignore the load that would add to the oplog and mongo DB 1000000 unnecessary updates every 30 seconds, yikes.

“resume when the server has capacity” also means exactly that - when the CPU usage of the server drops below a certain threshold, resume - this way when the server is idle the job runs as fast as it can, when the server is busy, the job runs more slowly - but the server is still responsive. The purpose of this is not for REALLY long running jobs that should run in their own process, but jobs that take 2-3 seconds, one or two users running that job, no big deal, 20 or 30 users running that job… This is a bigger problem, but potentially not worth spinning up a new server, or new process to handle.

It is not just CPU usage I am interested in - DB calls is another metric of interest, throttling DDP invocations only goes so far, depending on the arguments issued a DDP call could return 1, 50, 5000 documents, it could return just 1, but require 50 or 1000 documents to get the information required by the one.

“If you want to like, idle more” - yes, I do, but I want to Idle more by an amount that changes, based on usage, using values that I don’t want to rewrite/redefine in every function where I want to use it - one option here is abstracting this type of functionality to a package…

19

Sure, this is really easy.

var Fiber = require('fibers');

function sleep(ms) {
	var fiber = Fiber.current;
	setTimeout(function() {
		fiber.run();
	}, ms);
	Fiber.yield();
}

Meteor.methods({
longFunction() {
    let jobId = Jobs.insert({progress: 0});
    Meteor.defer(() => {
      let res = 0;
      for (let i = 0; i <= 1000000000; i++) {
        res += Math.random() * Math.random();
        if (i % 1000 == 0) {
            // This will yield in the way you need it to.
            Jobs.update(jobId, {$set: {progress: i}});
            // Now idle
            sleep(Jobs.count()*1000);
        }
      }
      Jobs.remove(jobId);
    });
    return jobId;
});
20

You’ve misunderstood me - I was just confirming that your code would pause for a few ms.

I’ll try once more - at build time, I do not know how long I will need to yield for. I will only know that at the time the method is called, and each time it is called, the amount may change - depending on how much load the server is under.

All I’m suggesting is writing functionality that will look at the current CPU load, and determine
a) if I should yield and
b) when I should resume.

21

The code could pause for a few ms, or it could pause for hours! Again, it’ll do work in between. (In practice, it won’t pause for hours, but it will do what has “queued up” to get done, and that’s exactly why you won’t experience delays at 100% CPU, so long as you “yield” frequently enough).

It’ll be so much easier for you to wrap your head around this if you instead delay running the job based on job count, instead of looking at the CPU.

22

I don’t care about the number of jobs - you’re stuck thinking this is a one off, or that the count of jobs will give me some information, it wont. The job count might be 1, I don’t care how many jobs are running - I care how much resource a job is using.

23

As long as you yield frequently enough, which is as easy as a heuristic as it gets, it will just work. It doesn’t matter what the load is. That’s what the operating system does, it doesn’t yield more or less depending on CPU frequency, that is what process/thread priorities for threads boils down to.

I don’t know what kind of scheduler node-fibers uses, but my guess would be it’s a fair one for IO, so it will probably just work.

24

“Fair” is subjective here - fair would be giving every fiber the same amount of CPU - fair is not useful if you have a time critical function, and a non time critical function.