This looks familiar to what @radekmie is going for with DDP Router:
I am not aware if anyone is pursuing this on their own.
I’ve been researching real-time and performance for at least a few weeks, and as I understand it better, I think we need to revamp the architecture a bit, from researching turning key code into Rust/WASM, worker threads, or even a separate queue for real-time changes.
Right now, all document updates are processed eagerly, which results into things like “oplog flooding”. If we had a queue somewhere, I think it would be fine for it to delay execution a few seconds if there is a peak of updates. Perhaps even having some kind of back-tracking or analysis for dropping repeated operations, away from the main Node.js process, perhaps even written in Rust for speed.
Like DDP Router but at the app level, away from limitations from Node.js/Async Hooks etc.
I am having all sorts of crazy ideas, perhaps one of them is valid. However I want to refactor the mongo package to TypeScript as much as possible and try the safest bets first before trying some of crazier ones.
The next logical step in our roadmap in improving real-time seems to be DDP batching, grouping as many added/updated/removed calls together as possible and prevent spamming so many microtasks.
Then we perhaps go to Change Streams, and after that we will need to stop playing safe and go to some of the crazier ideas.
I would love to chat/meet about any of this if anyone is up to it.