Wow, that’s a lot of feedback! Thank you all! I let myself comment on all of them in one comment not to spam too much. I’ll also skip some parts of your comments if the answer is already there.
@minhna
You can load the documents via methods then insert/update them to/in minimongo. Then other parts will be the same.
That’d require a lot of client-side changes to make sure that all of the code is using the local, method-populated collection. Again, an interesting idea, though a hard one to implement in a huge app.
@vooteles
Interesting topic. You might want to give this package a try:adtribute/pub-sub-lite
I saw that some time ago. It was also proposed by @jkuester. The problem is that it does not support any real-time updates. I may, however, experiment with pooling using this method.
@renanccastro
I did this package some time ago aiming to solve this exact same issue.
I’ll definitely try it if we’ll ever use redis-oplog!
- if you send a single very large message, the server can hang while it processes it […]
- garbage collection […]
- BSON parsing […]
- We have the exact opposite problem - we do send a lot of small messages. There are some large ones, but they do not correlate with the CPU spikes at all.
- We are at ~12% (~1GB) of RAM almost constantly. That’s far under the default limit (~1.3GB) and we do use
--max-old-space-size=3072anyway. - It is the case with these larger messages from 1., originating from our extensive usage of aggregations. Again, it does not correlate with the CPU spikes.
@don1231
Here’s a bit of a crazy idea. Not sure if this will work since I’ve never tested this.
I’m not sure how to implement it at all. Meteor has to populate the merge box on the server anyway and it’s probably not possible to do it from the client side (the already downloaded data). If you’d have an idea on how to do that - let me know! (We’re hosted on Galaxy - it may be a little problem with separate cluster.)
@efrancis
We really would need to see your Meteor APM or Monti APM info to help debug this type of thing. […]
Does one user logging in slow the app for all users? […]
What is your data model? […] If you could share your full data model, […].
As a side note, Meteor likes big containers because CPU usage can be high, so I’ve found that a few large servers performs better than a lot of small servers.
- Of course, I’ll attach them at the end.
- It doesn’t make the app slow at all. The app is unresponsive at all, for a couple seconds. And no, not for all users, but for all on the same container.
- It’s highly normalized with occasional denormalization of rarely changing data. I can’t share the full data model because of the business agreements, as well as the fact that’s far too large to analyze here (>60 collections).
- We are currently using Galaxy’s Octa servers (8.3 ECU) - these are already quite strong. However, I do agree with the comments from Galaxy performance vs AWS - it feels like these Galaxy units are somehow limited.
@macrozone
[…] Is it because a lot of users log in at the same time or because a lot of users subscribe at the same time to a lot of documents? […]
[…] Careful when updating a lot of documents at the same time regularly. […] Solution is to usedisableOplog: trueon subscriptions that need this data. […]
- Neither. The problem occurs when certain users (with relatively high number of documents) log in.
- We’ve never had problems with that as the number of inserts/updates/deletes in our case is relatively low.
- I’ve tried enabling
disableOplog: truefor all publications. It did not affect the loading time in anyway, both on client and server.
Here’s a couple of APM screens (highly censored, sorry).
