DDOS Meteor issue

#1

We have 6 Meteor instances in production that runs behind nginx on one server. And MongoDB that is running on the other server. The issues occur when a lot of changes (about 5000 per second) happen to the MongoDB. All meteor instances start to receive a huge amount of data by the network (speed up to 300 Mbps). CPU raises up to 100% and meteor dies until changes to MongoDB stop happening. That happens even when there are no active publications/subscriptions. Looks like it’s happening just because of DB updates. Also, we have 2 other meteor apps and they are connected to the same DB. What’s interesting, they also receive some unknown data with the enormous speed.

As far, as we understand it happens because of reading oplog. We are looking for a workaround because every time it happens it causes DDOS.

#2

Have you checked the forums? For example:

#3

Can you check if your Node version is okay for you Meteor version?

#4

Thanks for the clue redis-oplog and n direction to move on.

Started integration of redis-oplog and now, we are facing another issue.

We have an ‘events.files’ collection, that has thousands of inserts per second.
MongoDB is being updated by Node.JS. We are manually sending a change to the Redis from an insert callback. The documents all have the same taskId field, which is a Mongo object.

We tried to do it the following way and we see the event in a debug mode of redis-oplog.
But the collection is not reactive and re-subscription doesn’t happen on Meteor side🙁

redis.publish('events.files', JSON.stringify({
 e: 'i',
 d: {task: taskId}
}));

What are we doing wrong? Please help!

#5

Node version is fine 8.9.4.

#6

Are you using the same Redis instance for your node and Meteor apps?

#7

Yes, we have just one instance of Redis. It is the same because, we see events in debug mode of redis-oplog in console, when inserting documents from Node. But the re-subscription doesn’t happen in Meteor. It’s not reactive.

#8

Do you use rate limiting? Do you use Meteor methods and/or correct permissions for insert/update operations? What happens if you do while(true) yourCollection.insert({some data}); in browser console? Is such ddos attack possible on your app by a malicious user?

#9

Are you using .observe or .observeChanges on any of your collections? ( https://docs.meteor.com/api/collections.html#Mongo-Cursor-observe). These methods create a live query and can be quite resource intensive.

#10

@ignl Yes, we do. Such attack is not possible, of course.
@ninjaPixel Yes, we do. They are allright. Tried disabling them, didn’t help.

The situation happens on a big amount of inserts to any of the collections. Meteor cpu raises a lot.

Now implemented a solution on tulip/oplogtoredis and cult-of-coders/redis-oplog package. Things became a little better, at least CPU lowered to 70-75% instead of 100% load while doing inserts and network exchange raised a bit. To remind we have 8 core processor with 6 Meteor instances behind ngnix. Looks like the problem is still there :frowning:

Before when we used Mongo Oplog it was like that.

image image

Now, after we implemented Redis solutions thing became different.

image image

In debug mode of redis-oplog, we see oplog events coming not depending on a query in publications on Meteor. That looks weird. Because we want to process only info that we need (queried), don’t we. I wonder whether is it a way it really should work?

Also, I wonder does it matter where to deploy tulip/oplogtoredis. We made it on MongoDB server. And Meteor is installed on the other server. May be it is worth installing it to Meteor server?

@diaconutheodor, may be you might help too?

Thanks you all, guys!