100 users on 16 core VM
Meteor (master x1) 27 - 39% (lots of spikes 50 - 63%) -- 250MB
Meteor (clients x2) 9 - 11%
Mongo 7-9% (spikes of 13%)
Meteor (master x1) 24% -- 200MB
Meteor (clients x2) 11%
Here is our test scenario, and it's related to our app:
We have a master (the teacher) and 100 clients (student)
Our app sends data from teacher to student and back. The most stressful case is when the teacher has many students. The teacher continuously monitors all students, their screens, what they are doing, live in the classroom.
CPU-wise and network-wise this is a serious stress for any system as it's continuous feedback.
Following conversation with @diaconutheodor I thought I would mention this. All our publications are _id - based. Which is the most optimal way for a cursor even with oplog. However, you will still get degradation of performance as you scale up.
We are running the tests off a much more powerful machine than before (16 vs 4 cores). So the Meteor clients are responding faster to requests, which are batched and spread out to reduce server loads. I think this is why we are seeing less of a difference than in prior scenarios. However, this makes sense, as we only have 101 users.
The spikes in this test case are more evident and more pronounced. If we had many classrooms running, these spikes would become the average and would bring our application to a halt. This is with 100 students only, imagine if we had 500 or a thousand. So for us, this package is a must as we are dealing with live data where oplog trailing will simply grow exponentially in CPU consumption. Also, the mongo DB spike is concerning, even if you outsource your DB (as you purchase DB CPU's usually) as it will delay your data exchange.
Improvements in Our App
During the testing, much of the data we are sending does not really need to go to the DB. If we properly implemented channels that do not have to go Mongo, I expect even further improvements.
For any real-time app with many users and live data, I can't see a way around Oplog-Redis, even if you implement GraphQL as the problem is structural (MongoDB does not have pub / sub). Mongo Oplog trailing will result in degradation on all Meteor clients as new instances come up and more users log in.