I’m in direct discussion with Nathan who is the maker of the package and he advised me to use it like this. Actually will have a call with him tomorrow so I’ll have more insight. I may not use it fully properly which is another problematic but the process is pretty smooth actually.
The jobs is a specific collection in mongodb,
it has some needed info for the logic and that’s it.
I’m looping through the jobs to do my action. When a job is completed I delete it from the collection.
So generally the jobs collection doesn’t have more than 20 or 30 items., it’s very fast to loop through. actually everything was working smoothly except the refresh so maybe with redis:oplog we’ll be good to go.
Sorry I reread the thread more carefully. And your issue with the observer latency as you pointed out.
And echoing what others stated, without oplog the default would be polling which add latency. And there could be locking/delay happening at the DB as well. Both issues can be by passed using Redis oplog, Vent or streamer. Sorry again I was just trying to see if there was a better approach first before optimizing the current one.
Don’t be sorry, you raised very interesting points. Thanks a lot for taking the time.
Oplog was clearly an issue. We still have some load issues using the cluster library but hopefully will be worked out in the following days. There surely also are processes that can be improved in our approach and I appreciated your brainstorming!
I am sure there are different way in achieving the final solution, as usual in coding, and it’s nice to have input from others on how they would deal with it.
Allows you to execute a group of queries while having a lock on the underlying documents (as @alawi mentioned above). It removes your problem of simultaneous updates and finds overlapping from different sessions
