we are using as solution for meteorjs the galaxy server with mongo cloud. The solution is pretty easy for deployment etc. But unfortunately we experience lag and delay on the server, which is much bigger than the delay in our localhost server. We also tryed to test the different meteor calls and publications with meteor-down, the data provided shows on the galaxy server up to 10 times slower response time than the localhost. So we have tryed to connect the localhost with the mongocloud - and there we have the same slow speed like on galaxy.
We updated already mongodb on mongo cloud to the version 4.0.x (latest stable release). We also put indexes everywhere where possible. We are using the Atlas M10 Instance - AWS (AWS cloud provider)
So our concern is regarding mongodb cloud with clusters. Somehow it is too slow and we don’t know how to make it faster. We are thinking that we are not the only one who is using this, but why we experience such lag in the mongodb connection? How to make it work faster? Do you have any suggestions?
You should probably make sure that the both Mongo and Meteor servers reside in the same AWS region. When the servers are physically close to each other the lag is usually smaller.
If it doesn’t work for you we (Astraload perf team) can weigh in and solve your performance issues this or another way.
Beside what @eluck has said M10 is one of the smallest instances, for DB. So if you have large queries or larger load you might want to consider upgrading it to a larger instance with more resources. Also check that you have properly configured oplog. Another option to increase performance would be adding Reddis Oplog, but that is a scaling improvement (guessing that you use M10 that is not the issue here).
To me it sounds like the physical distance between the DB and Galaxy is the issue. I’d double check to make sure both reside in the same region/datacenter.
I thought redig-oplog was more of a scaling tool to reduce the impact of high load on app servers by replacing oplog tailing with a more fine grained system based on Redis. I’m not sure it can help you, when at the end of the day, you still have to fetch the data from the DB sitting on the opposite side of the planet (…in the worst case scenario).
@drew Thanks for your advice. We setup an EC2 instance and installed redis as described. However it doesn’t seem to improve our average pub/sub or method response times in the APM stats. I am running a meteor-down test of 100 simultaneous users to load our site. Any thoughts?
I had to do a lot of research on the forums to setup redis-oplog correctly.
In .meteor/packages ensure cultofcoders:redis-oplog and disable-oplog are listed on top of all the other third-party installed meteor packages (order matters).
Have you also ensured that a connection is being made in console on server run? It should print a message with Redis connected or similar on default settings.
@drew see this discussion here https://github.com/cult-of-coders/redis-oplog/issues/339
From what I understand, redis-oplog is only meant to improve scalabilty, i.e when using multiple meteor instances. It is not expected to improve DB response times on a single instance.
Which seems to contradict what you said earlier in this discussion:
Implemented redis-oplog (made the most impact)
I am wondering why you saw so much improvement on your system. Are you using multiple instances?
We use a sharded cluster (across 3 geographic regions) - which does not support oplog tailing easily. We had huge latency as we waited for each action to propagate across each geographic region (i.e. London to Sydney to US). By replacing it with redis-oplog as a centralised source, we saw huge performance increases. This allowed us to support roughly 600 concurrent connections without degrading our performance (where methods & pubsubs were unusably slow).
You may be able to test modifying subscription polling times as listed here: https://docs.meteor.com/api/collections.html . Before we implemented redis-oplog, editing polling subscription times (i.e. the time before a subscription would ‘refresh’ once new data) made a little performance improvement on our subscriptions. But this was due to the same fundamental problem of oplog tailing across a sharded cluster.
