Meteor Scaling - Redis Oplog [Status: Prod ready]


#546

@diaconutheodor , CPU is spiking with redis-oplog in place.
We have two microservices.
1st Microservice is master and core service and
2nd microservice is background job processor (which does highly cpu intensive computation and bulk inserts/updates).

Without redis-oplog only cpu used to spike at 2nd Service which does bulk inserts and update on single collection.
Since we want our system to be reactive. Hence forth we enabled redis-oplog for both the services. And now comes the problem.
When there is JOB (bulk updates) proccessing in 2nd microservice the CPU is also spiking at 1st microservice.
When I disable redis-oplog in 1st microservice everything works smooth except we loose reactivity for the operation that are done at service 2.
I enabled debug for Redis-oplog at Service 1 and found that it is writing continously below logs
[RedisSubscriptionManager] Received event: “i” to collectionName
[RedisSubscriptionManager] Received event: “u” to collectionName
[RedisSubscriptionManager] Received event: “u” to collectionName
… etc

Thanks,
Koti


#547

@koticomake you head into the same problems as with mongodb oplog, your instance gets flooded with tons of information. RedisOplog without being fine-tuned is ultra fast with publications by _id and just a tiny bit faster with standard mongodb oplog. (Maybe slower in some cases). Where it shines is in it’s ability to control the reactivity.

Questions:

  1. Your CPU does not spike if you’re tailing the oplog ? Are you sure you are tailing the oplog and not relying to polling ? Did you test this in prod ? If yes, do you have MONGO_OPLOG_URL set ?
  2. Can your publications be fine-tuned ? Maybe namespaced by a client or something ?
  3. Do you need reactivity at every step inside the job service ? Is the same document updated multiple times ?
  4. I can add a hack for you to do something like “trigger reload for certain collections” on the redisoplog cluster. That would be an interesting idea. To say something like, hey I finished my heavy batch processing, now I want everyone to reload.

#548

@diaconutheodor
1A. We are not using MONGO_OPLOG_URL and moreover we have included disable-oplog package as well in to our meteor project. Do we still need to use MONGO_OPLOG_URL ? to make redis-oplog work ???
2A. I will try to fine tune our publications with NameSpace.
3A. Reactivity is not required at all steps inside of our JOB service. All inserts can have direct reactive but all updates can be reactive once the batch operation is done. And yes, Same document might get updated multiple times as well.

I am really excited to get the hack that you promised at point 4.

One more doubt. I might be dumb asking this.
How come redis-oplog (RedisSubscriptionManager) events effect’s main server’s cpu ?? will this not directly deal with Mongo DB operations ?

Thanks,
Koti


#549
  1. That’s what I thought. You didn’t tail the oplog, you were previously relying on polling. If you would have had mongodb oplog enabled, CPU spikes would have been a bigger issue
  2. Perfect, that would really boost performance
  3. Perfect, you have the option {pushToRedis: false} only do the update once at the end and push it to redis.
  4. It affected the “Main Server” because you had a publication listening to messages on that collection. My guess is that you have something like:
Meteor.subscribe({
    items: () { return Items.find(someFilters) }
});

^ That subscription alone, regardless of filters (unless they are by _id) will listen to ALL incoming redis messages, derived from operations performed on Items collection. (Unless you namespaced it)

The true value of RedisOplog lies in fine-tuning, publications by _id and ability to perform writes to db and bypass reactivity. That’s the true value.


#550

Congrats on surpassing 1000 downloads :slight_smile:


#551

@diaconutheodor,
I have one doubt here
3. Perfect, you have the option {pushToRedis: false} only do the update once at the end and push it to redis.

When I update collection in my second service. I kepts for all intermediate updates PushToRedis as false. For all insert I kept pushToRedis:true.
Now how shall I push all changes that happend on that collection to redis ??
Do I need to use this below code and if yes, Do i need to push each and every record by _id ?? Is there any way that I push all changes at once to redis once I am done with my JOB ?

getRedisPusher.publish('tasks', EJSON.stringify({
    [RedisPipe.DOC]: {_id: taskId},
    [RedisPipe.EVENT]: Events.UPDATE,
    [RedisPipe.FIELDS]: ['status']
});

One more,
The true value of RedisOplog lies in fine-tuning, publications by _id and ability to perform writes to db and bypass reactivity. That’s the true value.

I didn’t get your point when you say “ability to perform writes to db and bypass reactivity” ??
What exactly you mean when you say bypass the reactivity ?

Regards,
Koti


#552

ability to perform writes to db and bypass reactivity => { pushToRedis: false }

And regarding your idea, if you do the updated with {pushToRedis: true}, you don’t have to manually send the events, if you don’t then you have to. And unfortunatelly the redis npm driver provides no way to publish multiple messages in one go, so you’ll have to do it in a loop.


#553

I’m trying to see if anything can be done about this. I have the exact same redis-oplog Vent emission I need to send to 1000 Vent-subscribed clients. And redis-oplog is looping to do this, specifically the this.on block inside of a Vent.publish block. It seems like there must be a way to send one update to Redis and then have Redis loop out to the subscribers? You’re thinking it’s a limitation of the Redis NPM driver?


#554

When I update the value’s directly in DB they are not getting reflected with my subscription.
I was thinking that redis-oplog’s cache is not getting invalidated and updated.
How to deal with these kind of approaches ??


#555

This package works by placing a hook on the Mongo functions of Meteor. Therefore, for this to work, you must call the Mongo functions. Editing directly the db obviously won’t call the necessary hooks.


#556

https://github.com/cult-of-coders/redis-oplog/blob/master/docs/outside_mutations.md

There’s also another package that’s been developed to deal with this too… it might be mentioned in this thread. I’ve heard @diaconutheodor talk about it somewhere.