Out Of Sync: Meteor Writing to MongoDB But Not Reading -- Oplogger Issues or Failure?



I’m wondering if anyone has had issues with MongoDB apparently getting out of sync with their Meteor app. We’re currently running a set up with around 15 production app web heads.
On the back end we have a 5 headed MongoDB replica set. 1 master write head, 4 read heads.

The issue we’re running into is that periodically (about every two weeks) we’ll have an incident where writes are going through to the database, but limited or inconsistent reads come out on the other end. What this looks like from a user perspective is that I would update a Todo – the todo would be updated in the actual database – but the update would not make its way out to the UI of users.

Restarting the app or restarting the DB has fixed the issue temporarily. That’s about all I know and I’m not sure where to look next.

I know that the Oplogger is used to determine changes and updates for Meteor. Is this a good place to start looking? Has anyone run into issues similar to this one? We’re also currently still on 1.1 for production, while our MongoDB is on 3.2.

Any ideas or experiences you can share are much appreciated.