Needed some help or advice as I am not sure how internals of Redis Oplog work.
We are using Redis Oplog in production. Sometimes there is a need to change some data directly on Mongodb on production using Robomongo. But I have noticed many times the changes do not appear/flow to the app as soon as the changes on database/collection is done. It is clear that data is cached in Redis and the change in database outside of app is not sensed.
Is there a way to force the flow of data for backend changes?
Thanks …I do understand now what we will have to do if outside mutations are done. But I was interested in knowing where there is a need for quick fix scenario – like deleting a record, or changing a field value using mongo shell or any tool and then forcing the flow – any commands using redis-cli, for example, to make it happen (restart of redis seems to work but I do not think it is right to do like that)
Cannot help on that front as my knowledge of redis-oplog is limited to what is on the documentation.
Regarding restarting redis, not sure why that is working, but maybe the events are being “reactivated” when reconnected to the stream. But yes, seems a very drastic solution as it will affect the entire application relying on that redis instance
But to think of it, if you are already using mongodb shell, then if you are not using a custom namespace (as indicated in the fine-tuning section), then you can just call a redis command from the CLI to publish an event (as indicated in the outside mutations)
This is issue I am running into, because I do not have access to meteor shell on galaxy sometimes I need to change data directly from a GUI so I end up having desynchronised datas… @perumalkuk Did you manage to restart redis from node process ?
Consider deploying my fork of oplogtoredis alongside your redis-oplog-enabled Meteor app.
Oplogtoredis is a standalone Golang application that continually tails the MongoDB oplog and publishes changes to Redis.
The original version of oplogtoredis was created by a company named Tulip, but last year I engaged the original programmer Ben Weissmann to update it to use MongoDB’s official Golang driver, make it compatible with the MongoDB 5.0 oplog format and support privately issued TLS certificates.
Not only does using oplogtoredis provide improved performance, but it also ensures that changes to the MongoDB collection made outside of your app are captured and pushed to Redis.
To make redis-oplog work with oplogtoredis, you need to set the redis-oplog configuration option externalRedisPublisher to true and optimistic to false. Here is an example:
I think that you may be the best reachable person to answer my question asked in Issues · tulip/oplogtoredis · GitHub what actually happened in the v2 of oplogtoredis? We’ve been using it for quite some while now and had no issues with it (we’re still on MongoDB 4.4), but I’m hesitant to switch to v2 without any information regarding the changes. Similarly, your fork has some improvements but also misses 27 commits from the original repo now.
I also would like to remind @storyteller that I’m still waiting for this
What are the advantages of your package? If you read from the oplog, why shouldn’t I use the standard implementation of Meteor which also reads changes from the oplog? The advantage from redis-oplog is, that you can decide by yourself, which updates should be reflected by the application. For example, if you do a heavy batch remove to delete old data from your collection, you’ll run into CPU issues since Meteor has to check each removed document for relevance.