Thanks for taking the initiative. As mentioned by many here, this is the biggest pain point. I initially thought about using RethinkDB, but (aside from the fact that they crashed recently) their engine was immature in some aspects (e.g. indexing) not to mention their query language.
Your post got me thinking. The reason oplog does not scale well, is that ALL meteor apps (on that DB of course) have to watch ALL the oplog (to compound the issue, if you don’t have the id of that collection in your query, a second mongo call may have to be made to get it). What if we had an intermediate firmware, that watched the oplog (i.e. only one oplog monitor per cluster of meteor instances) and we simply registered to event handlers on it. In other words we distribute the oplog monitoring. Your solution is probably similar to that.
Would we still need Redis here (trying to avoid adding more db’s / data stores)?
We could even build this monitor in Meteor itself … again, just thinking, maybe I am missing something.