Redis oplog flow of data

perumalkuk · February 8, 2021, 5:41am

Needed some help or advice as I am not sure how internals of Redis Oplog work.

We are using Redis Oplog in production. Sometimes there is a need to change some data directly on Mongodb on production using Robomongo. But I have noticed many times the changes do not appear/flow to the app as soon as the changes on database/collection is done. It is clear that data is cached in Redis and the change in database outside of app is not sensed.

Is there a way to force the flow of data for backend changes?

Thanks

rjdavid · February 8, 2021, 5:51am

The fine-tuning section can help you understand how to update Redis with events

github.com

cult-of-coders/redis-oplog/blob/master/docs/finetuning.md

## Fine-Tuning

Now that we have a generic idea of how this works, we can make some fine-tuning that will **dramatically** increase the speed of your highly loaded app.

### Custom Channels

A very common scenario where this can be used is a chat application. Usually when we create a chat app, we have a parent "Thread",
and each message will have threadId pointing to our thread.

```js
Meteor.publish('messages_by_thread', function (threadId) {
    // perform additional security checks here

    return Messages.find(selector, {
        channel: 'threads::' + threadId + '::messages' // you can use any conventions that you like for naming them, it's not relevant
        // any name you choose just make sure it cannot conflict with a collection (threads), or a direct listening (threads::threadId)
    });
})
```

This file has been truncated. show original

When something is changed directly from the DB, it will not cascade automatically because no events are fired. You will need to do an outside mutation yourself

github.com

cult-of-coders/redis-oplog/blob/master/docs/outside_mutations.md

## Outside Mutations

There may be scenarios where you have an external service that is not Meteor, which
performs mutations to MongoDB. If you want your subscriptions to "catch" the changes you have to send 
manually the changes to redis channels.

However, this is pretty straight forward.

Given I have the following publication:
```js
return Tasks.find({groupId});
```

If an outside worker updates the MongoDB, it should also send changes to redis like this:

```js
// update task document, that changes status inside MongoDB
redis.publish('tasks', JSON.stringify({
    e: 'u',
    d: {_id: taskId},

This file has been truncated. show original

For us, when there is “DB migration” required, we always create a “DB script” which uses existing functions in our collection files which will trigger the redis-oplog reactivity

perumalkuk · February 8, 2021, 6:24am

Thanks …I do understand now what we will have to do if outside mutations are done. But I was interested in knowing where there is a need for quick fix scenario – like deleting a record, or changing a field value using mongo shell or any tool and then forcing the flow – any commands using redis-cli, for example, to make it happen (restart of redis seems to work but I do not think it is right to do like that)

rjdavid · February 8, 2021, 6:30am

Cannot help on that front as my knowledge of redis-oplog is limited to what is on the documentation.

Regarding restarting redis, not sure why that is working, but maybe the events are being “reactivated” when reconnected to the stream. But yes, seems a very drastic solution as it will affect the entire application relying on that redis instance

rjdavid · February 8, 2021, 6:32am

But to think of it, if you are already using mongodb shell, then if you are not using a custom namespace (as indicated in the fine-tuning section), then you can just call a redis command from the CLI to publish an event (as indicated in the outside mutations)

sabativi · August 22, 2022, 2:03pm

This is issue I am running into, because I do not have access to meteor shell on galaxy sometimes I need to change data directly from a GUI so I end up having desynchronised datas…
@perumalkuk Did you manage to restart redis from node process ?

vlasky · August 24, 2022, 2:39am

Consider deploying my fork of oplogtoredis alongside your redis-oplog-enabled Meteor app.

Oplogtoredis is a standalone Golang application that continually tails the MongoDB oplog and publishes changes to Redis.

The original version of oplogtoredis was created by a company named Tulip, but last year I engaged the original programmer Ben Weissmann to update it to use MongoDB’s official Golang driver, make it compatible with the MongoDB 5.0 oplog format and support privately issued TLS certificates.

Not only does using oplogtoredis provide improved performance, but it also ensures that changes to the MongoDB collection made outside of your app are captured and pushed to Redis.

To make redis-oplog work with oplogtoredis, you need to set the redis-oplog configuration option externalRedisPublisher to true and optimistic to false. Here is an example:

"redisOplog": {
    "redis": {
        "port": 6379,
        "host": "127.0.0.1"
    },
    "mutationDefaults": {
        "optimistic": false
    },
    "globalRedisPrefix": "meteorapp.",
    "externalRedisPublisher": true
}

I note that most redis-oplog fine tuning options won’t work when using it with oplogtoredis, but perhaps you will find that they might no longer be needed.

sabativi · August 24, 2022, 9:45am

Thanks for that, I do not use any options of redis oplog so it might works out of the box.

Will try on staging app first

Thanks

radekmie · August 27, 2022, 11:31pm

I think that you may be the best reachable person to answer my question asked in Issues · tulip/oplogtoredis · GitHub what actually happened in the v2 of oplogtoredis? We’ve been using it for quite some while now and had no issues with it (we’re still on MongoDB 4.4), but I’m hesitant to switch to v2 without any information regarding the changes. Similarly, your fork has some improvements but also misses 27 commits from the original repo now.

I also would like to remind @storyteller that I’m still waiting for this

XTA · August 31, 2022, 2:25am

What are the advantages of your package? If you read from the oplog, why shouldn’t I use the standard implementation of Meteor which also reads changes from the oplog? The advantage from redis-oplog is, that you can decide by yourself, which updates should be reflected by the application. For example, if you do a heavy batch remove to delete old data from your collection, you’ll run into CPU issues since Meteor has to check each removed document for relevance.