Confusions about DDPRateLimiter rules

core
1

The DDPRateLimiter package is a very powerful rate limiter that provides protection from DDOS-like attacks, and security against brute-force attacks and also adds processing limitations to features. It is very flexible but just like many things, this flexibility can result in complexities.

Since the documentation is not explicit about a number of things on rules creation, can someone help confirm if the expected behavior of the rules is correct as stated below? For reference here is the documentation for the addRule() function: Methods | Meteor API Docs

Global limits rule

const matcherRule = {
  type: 'method',
  name: 'resourceIntensiveProcess'
};

I believe this is where most of the confusion starts. When done like this, the rule is applied to ALL calls of the method. This means that all clients share the limits. So a client can be limited even the first time it calls the method.

Per registered user rule

const loginRule = {
  userId(userId) {
    const user = Meteor.users.findOne(userId);
    return user && user.type !== 'admin';
  },

  type: 'method',
  name: 'login'
};

This is the example in the documentation. This means that the limit applies per registered user and only for registered users. There are NO limits for unregistered users.

Per connection rule

const connectionRule = {
  type: 'method',
  name: 'perConnectionProcess',
  connectionId() { return true; }
};

This means that the limit only applies to each unique connection. If the client refreshes and a reconnection happens, the limit is also refreshed.

Per IP Address

const connectionRule = {
  type: 'method',
  name: 'perIPProcess',
  clientAddress() { return true; }
};

This implements a limit per IP address. It can be used to temporarily add a limit to specific IP addresses when needed e.g. hitting initial limits, so that other processing will not be affected

No limits

const connectionRule = {
  type: 'method',
  name: 'perIPProcess',
  clientAddress() { return false; }
};

I also just like to confirm if returning false on any of the expected keys will result in the matcher not matching and therefore resulting in no limits. Is this correct?

I hope someone who knows this better can give confirmation on the functionalities of the DDPRateLimiter

2

Referring to the following read me file.
https://github.com/meteor/meteor/blob/master/packages/rate-limit/README.md

It appears to me that your “No limit” rule will keep count of the total number of method calls for the “perIPProcess” method. And it will be only cleared when the intervalTime is passed.

The default intervalTime is 1000ms, And the default numRequests allowed per time interval is 10.
So in your case rate limiter will prevent more than 10 calls in total to the perIPProcess method.

Default values.
https://github.com/meteor/meteor/blob/master/packages/ddp-rate-limiter/ddp-rate-limiter.js#L67

3

[Update] Reading the readme file you posted, it says

A rule is only said to apply to a given input if every key in the matcher matches to the input values.

Therefore, if the return value is false on one of the keys, then there is NO match. The rule is not matched at all instead of treating as if that particular key does not exist

1 Like
4

Another fun quirk we just ran into is if you apply a global rule to all subscriptions (or methods) by just doing the following:

DDPRateLimiter.addRule({
  type: "subscription",  // Or "method"
  connectionId() { return true; },
}, 50, 3 * 1000);

It seems you then can no longer pass per-subscription or per-method rules after this (or before). We’re finding that if a global rule is set, it will always match to that.

We have a few special subscriptions that need to run more frequently than our global rule. It seems to fix this you’d have to create a rule for every subscription or method individually. Can anyone else confirm this?

5

[Update] I just realized you specified (after or before)

Each rule is independent. So it does not matter if there is a “global” rule or a “more specific” rule. If the rule matches the inputs, the rate limit rules will be used

1 Like
6

Right. I just put that to give context that we were troubleshooting the order to see if that made a difference.

It sounds like if you want any sort of “global” rule, then that will effectively override any per-function rules. :confused: Is there some clever way using underscore/lodash to like pass in a giant list of all your methods/subscriptions minus anything you want to be per-function? I think I saw an example of this. That you could pass an array of names, but then exclude certain functions that have per-function rules?

7

You could use Meteor.server.method_handlers to list all the methods and then use the name parameter to match them. Is that what you mean?

8

I mean does the rule matcher accept an array of names. Or would you have to loop it adding a rule for every method/subscription individually?

9

I loop over on startup… something like this goes in my projects. It tells me on startup if I have methods without a rate limiter and it tells me if I have a rate limiter set for a non-existent method.

I bucket methods into slow, fast, veryFast…

// https://docs.meteor.com/api/methods.html#ddpratelimiter
import _ from 'lodash';
import { DDPRateLimiter } from 'meteor/ddp-rate-limiter';
import { Accounts } from 'meteor/accounts-base';
import { Meteor } from 'meteor/meteor';


// clear existing rule
Meteor.startup(function () {
  Accounts.removeDefaultRateLimit();

  // get all the methods
  const methods = _.keys(Meteor.server.method_handlers).filter(function (each) {
    return each.charAt(0) !== '/';
  });

  const slow = [
    /* From meteor/accounts-base */
    'login', 'logout', 'getNewToken', 'removeOtherTokens', 'configureLoginService',

    /* From meteor/accounts-password */
    'changePassword', 'forgotPassword', 'resetPassword', 'verifyEmail', 'createUser',

    /* From meteor/accounts-passwordless */
    'requestLoginTokenForUser',

    /* Server methods */
    'mySlowMethodOne'

  ];

  const fast = ['myFastMethodOne'];

  const veryFast = ['myVeryFastMethodOne'];
  
  
  const all = slow.concat(fast, veryFast);

  all.forEach(function (each) {
    // check to see if we have unnecessary method names in the above arrays
    if (methods.indexOf(each) === -1) {
      console.log(`${each} is not an active method. Consider removing it from the rate limiters`);
    }
  });

  slow.forEach(function (each) {
    DDPRateLimiter.addRule({
      type: 'method',
      name: each
    }, 2, 1000, function (res, event) { // two per second
      if (!res.allowed) {
        //do something
        console.log(event);
        console.log(`The Rate Limiter was triggered with method: ${each}`);
      }
    });
  });

  fast.forEach(function (each) {
    DDPRateLimiter.addRule({
      type: 'method', name: each
    }, 5, 1000, function (res, event) { // 5 per second
      if (!res.allowed) {
        //do something
        console.log(event);
        console.log(`The Rate Limiter was triggered with method: ${each}`);
      }
    });
  });

  veryFast.forEach(function (each) {
    DDPRateLimiter.addRule({
      type: 'method', name: each
    }, 100, 1000, function (res, event) { // 100 per second
      if (!res.allowed) {
        //do something
        console.log(event);
        console.log(`The Rate Limiter was triggered with method: ${each}`);
      }
    });
  });


  const rules = DDPRateLimiter.printRules();
  // convert the rules object into array of method names with rules
  const ruleNamesArray = Object.keys(rules).map((i) => rules[i]._matchers.name);

  // if a method does not have a rate limiter - log the following message
  if (ruleNamesArray.length !== methods.length) {
    methods.forEach(function (each) {
      if (ruleNamesArray.indexOf(each) === -1) {
        console.log(`Method: ${each}' does not have a rate limiter enabled. Consider adding on at rateLimiter.ts`);
      }
    });
  }
});
1 Like
10

Whoa… this is awesome. Exactly what I was thinking/hoping for (and thought I read a forum/blog post a long time ago about doing something just like this).

Question - can you get all the subscriptions the same way? Don’t you rate limit those too?

11

Awesome, glad that helps. Maybe someone can improve the code.

I have not done it with subscriptions, I should though. Off the top of my head, I don’t know how to get a list of the registered subscriptions. But now that you mention it, I’ll take a look later and see if I can figure it out (unless some else chimes in here first).

12

Through some other posts I figured out that Meteor.server.publish_handlers can be used to get the list of subscriptions/publications. Also, be sure that your method and publication startup code runs before calling these DDPRateLimiter rules, otherwise they won’t be present at the time the DDP setup code runs.

BTW, @generalledger you may want to add connectionId() { return true; }, to your rules, or else these would be global rules across all users and exceed much more quickly and problematically no?

I ran into something of a road-block when trying to set up the above pattern in my app. I’m noticing that when you break everything down to per-method and per-subscription rules, the overall concept works differently than global rules. For individual rules, the threshold is per-method or per-subscription. Before when I had two global rules (one for methods, one for subscriptions) it limited all calls to all subscriptions/methods per connection. So you can set one large functional limit that is based on your app’s usage of methods and subscriptions.

I was incorrectly thinking that when doing the above “loop” pattern, they would still all “work together” for that threshold. But now each method and subscription has its own bucket threshold. Which now makes me have to rethink the global threshold I had and try to transpose it into an individual threshold.

For example, my previous global rule was:

 DDPRateLimiter.addRule({
    type: 'subscription',
    connectionId() { return true; },
  }, 50, 10 * 1000, ... { // 50 subscriptions every ten seconds

So this lets the user call up to fifty subscriptions every ten seconds. But if I apply this to a loop, like so:

allSubscriptions.forEach(function (subscription) {
  DDPRateLimiter.addRule({
    type: 'subscription',
    name: subscription,
    connectionId() { return true; },
  }, 50, 10 * 1000, ...  // 50 calls to THIS subscription every ten seconds

Now each one of my subscriptions can be called fifty times per ten seconds. Which is definitely not what I want. And I understand this, but to now have to break each individual subscription down to how many times it can be called in a given time frame is super cumbersome. As there are some that need to be called much more frequently than others.

I will say, it’s kind of clunky that Meteor didn’t write the DDPRateLimiter to give precedence (or have a priority mechanism) to per-method and per-publication rules. It seems logical that an individual rule would always have priority over a global rule, or else why add an individual rule? But instead you have to do this wild looping which, without several protection checks, you could easily leave open a gaping hole in your API. It would be so nice to just add two global rules and the few individual rules that you need to take precedence over the global.

1 Like
13

Had one more question about this line that you use to filter out all the method entries that are in the format:

/collection/insert
/collection/update
/collection/remove

What exactly are these? If I had to guess, it’s what Meteor adds to every collection to allow client-side transformations (that should be disabled due to security issues). So really those client-side transformations are just Method calls? :thinking:

And your pattern assumes and relies on them all being disabled, since you’re not adding rules for them:

Collection.deny({
  insert() { return true; },
  update() { return true; },
  remove() { return true; },
});
1 Like
14

Wow, this is going to be helpful. Appreciate the replies.

Yes, I always deny client side inserts. Way too risky to leave those for 99% of projects.

You are 100% correct about waiting for publication startup code to run before getting access to the list of publication handlers.

With regards to the connectionId, you might be right. I’ll do some tests this weekend.

And yes, I believe you are right about each limiter having its own bucket. That doesn’t bother me personally. I realize it isn’t quite the same, but for global rate limits, I’d just use a WAF. If you wanted a true global rule for methods, I supposed it could be done with a custom validatedMethod hook.

Maybe you could get close with a home rolled rate limiter using rawConnectionHandlers and Meteor.server.stream_server? Not sure. Might be worth a look.