Proposal: Rate limiting in Meteor Core

core
1

Hey everyone,

Anubhav here from MDG.

I’m working on adding DDP rate limiting to Meteor core. The DDP Rate Limiter can rate limit DDP method and subscription calls based on IP address, user id, and method/subscription name, and comes with default settings to limit login requests. Here’s a hackpad with examples and more info: https://meteor.hackpad.com/Proposal-Rate-Limiting-in-Meteor-core-HJafA26QtIy

Please provide comments on the hackpad and let me know if this satisfies your needs!

Thanks in advance,
Anubhav

14 Likes
2

Shouldn’t be part of meteor core if you ask me. In a typical org, rate limiters will be typically maintained by the operations department - and they don’t like to touch code. Not a good idea to mix dev/ops domains. my 2c, cheers.

4 Likes
3

I agree. Ratelimiting should be able to control via purely on settings.
Either using env. vars of Meteor settings.

But I like to have a core package, but avoid the use of custom code to make settings. I mean, don’t use functions like this:

var rule = {
    userId: null,
    ipAddr: function (ipAddr) {
        return true;
    },
    type: 'method',
    name: 'testMethod'
}
4

I think most Meteor users ( arguably ) don’t work on a “typical org”
:blush:

2 Likes
5

In principal, I would agree - separate ops / config from code - however, in the context of Meteor, I would disagree, because of:

  1. When you write a method/publication, you’ll have the clearest idea of the maximum rate it should be called at.
    For Example -
    • Writing a method used in an autocomplete search - max rate is 50 per 1000ms
    • Writing a login method - max rate may be once every 5000ms
  2. Packages authors will want to design their package around sane rate-limits; In most cases, they should be set by the package - not the application.

@arunoda - I’m sure there will be users who would like to rate-limit their own methods via environment variables. There could easily be a community package which translates them in to the necessary calls to addRule.

4 Likes
6

Yep I agree. That’s why I agree on the core package, but removing the use of callbacks in the API.

  1. I don’t think developers know about the usage of their DDP methods, until they deploy it into production. We can’t predict usage patterns.

  2. I don’t think so. Packages should not do it. Package authors also don’t know about it.

Rate limiting should be control from a single place.

1 Like
7

I think you could pick per-user soft & hard limits, within an order of magnitude. Enough to prevent heavy abuse, with the goal being normal users won’t approach these limits.

For example, you would know when writing/testing a “delete project” feature that the method deleteProject shouldn’t be called more than once per 100ms. Even though, performing the action in the interface takes 1.5s.

For that to happen, I’d have to have a list of every method called in my whole application, by every package, and then know what the usage characteristics of each are, and what acceptable limits should be set.

For example, what if package XYZ has deleteThingy function, and I set the limit to 100ms.

What I don’t realise that when you do a mass-delete, it calls deleteThingy many times from the client. (Instead of having a deleteManyThingies method).

If the package author set reasonable limits, this would be avoided (but still having the advantage of security / crash prevention). Perhaps the package author would have been cognisant of the limits set, and added a deleteManyThingies method.

2 Likes
8

This is an excellent feature to add! I’ve been adding this manually. It’d be convenient to have this out of the box.

9

I totally agree with your @nathan_muir. It should definiltley be programmatic, and yes, package authors (the ones who care) will love to know about it. If I make a search field with type-ahead (search suggestions/autocomplete) the end user may not know about or care about performance, they just want it to work. There’s plenty of scenarios where package authors would want to use this.

10

Okay. That’s a valid point. I agree.

11

Well, I think you are pointing at the issue right there:

“When you write a method/publication, you’ll have the clearest idea of the maximum rate it should be called at.”

I am afraid this is not a right statement - the ops guy is the only person who has visibility about the real traffic - the dev is only doing guess work. These two roles are typically different - not the same skill sets, not the same accesses, not the same stress level, not the same SLA.

Maintaining SLA/SLD in meteor can turn into a maintenance nightmare as the application complexifies. Add user/group/client based limits to the picture and you will start to get the picture.

My 2c,
Cheers,

12

I agree with @nathan_muir that we can create a community package to translate the necessary calls to addRule from environment variables.

Also, the use of callbacks is useful in cases where we need code to determine whether the rule should apply. For example, we may want to apply a rule to all users but admins. In this case, we could have a rule like this assuming we had a getUserType method available:

var rule =  {
    userId: function (userId) {
        // Rate limit all users but admins
        if (getUserType(userId) !== 'admin') 
             return true;
   }, ...
}

I also added this example to the Hackpad to showcase why we would want to be able to set rules using callbacks.

There is a good point brought up about who should know how to set rate limits and I’ve filed an issue with this, as there seems to be a need to know what stats look like on current DDP method calls before adding limits into current applications. Please move all discussions about the topic to that Github issue! Thanks for the great feedback so far!