I’ve been working on our app in Meteor since 2013 and have spent years optimizing, refactoring, etc. Presently, our app’s pub/sub and Meteor method usage is highly customized and evolved to be very use-case specific. We use redis-oplog, rely heavily on its Vent functionality, lots of caching, and Mongo bulk inserts (with redis-oplog’s ability to then notify Meteor about outside mutations).
Everything works pretty good, but it’s a lot more involved and more code than core Meteor. And I’ve always wanted to get off of redis-oplog and get back to oplog tailing. I’ve noticed that some users seem to use it successfully. As long as their code is heavily optimized I assume.
I’m wondering if these newer publication strategies could allow us to go back to non-Redis oplog and simplify our code base.
One of the largest hurdles we have is anywhere from 1000 - 5000 simultaneous users (or often more) all hit the app at the same time (think real-time game show). They all access the same publication and Mongo document (e.g. a game document). This needs to be reactive, but it’s literally 100% cursor reuse for all the participants.
Back in the day before publication strategies, it was disclosed that the server kept a copy of each user’s subscription data in memory, regardless of publication cursor reuse. Thus quickly scaling the RAM to the ceiling on each server. It seems to me that an option like
NO_MERGE_NO_HISTORY would work great for a single document publication where all we care about is changes to fields (e.g. game states). And less focus on a list of documents being added and removed. Does this publication strategy thus eliminate the RAM overhead?
Our current approach relies heavily on cached Meteor methods that debounce the same “game document” to each participant (thus bypassing Mongo for the 99% majority) but then annoyingly, albeit effective, relying on Redis Vent calls to keep reactive data in sync.
Anyone have any thoughts or input on the above? Meteor methods are nice because they can be cached and don’t cause a lot of overhead on the server, but they’re not reactive. Pub/sub is reactive, but historically had too much overhead.
The second issue is the flood of participation. Presently, we accept participant responses in a method that collects the responses on the server and only bulk updates them to Mongo on a rolling interval. Then we notify Meteor of the bulk updates (that are actually outside mutations at this point) using Vent.
It would be nice to continue using this approach, but without Redis Vent, is there anyway to update Meteor about outside mutations. As I’m betting bulk inserts don’t jive with native Meteor oplog tailing.