Wanted to share some more stats on my DDP
Ive got 20 active players on the server, and you can see 200 posts changed in basically 1 second.
I might have to make a que system for pushing updates to get that quantity down? Any suggestions are welcome.
If you want to see yourself, come check out www.StarCommanderOnline.com and check yourself.
Have you tried redis-oplog to speed up mongo?
BTW, redis way is storing data in a more simplified way(and type).
His problems are on the client though, faster server wonât help with that.
1[quote=âgothicmage, post:22, topic:36521â]
oplog
[/quote]
The MongoDB is now using OpLog on a 3 cluster system. It DRAMATICALLY improved things.
And now the client is bloated lol.
Doing some testing on it, Iâm sure Ill figure it out. Buncha smart folks here 
So Redis⌠Now that I have my OpLog server running - do I not need that with Redis? I looks like Redis is yet another layer I could add for performance and probably need in this case. Iâll look in to it.
More advise on it is greatly appreciated.
A lot of it has to do with the Minimongo library, the pub/sub subscription, whether there are classes being created, which rendering library youâre using, and how many components youâve created.
The browser can certainly handle many thousands of JSON objects⌠just ask anybody who does mapping or graphing. But to cleanly render those objects, you need a functional programming library that uses pure functions. Blazeâs rerenders just donât cut it. But D3 and React can.
So, if youâre just using D3 or React, itâs clearly easy to have an array of 1000 or 10000 JSON objects, and the browser renders super fast. So why does Minimongo bog down? Well, itâs monitoring the subscription and handling optimistic UI. For every component you create, there will be a Tracker item put on the heap thatâs watching to be invalidated. Itâs all those Tracker objects that bring it down an order of magnitude.
You can use HTTP.get() or Meteor.call() functions to bypass the Minimongo cursor, and load up thousands of records client side; but youâll loose reactivity and synchronization with the database.
tl;dr - your max of 225 documents isnât simply a max of documents in the browser. Itâs the max number of synchronized documents that will be synced with the database.
YES. This is exactly what Iâm trying to describe.
So what are my solutions?
Ahh I see. Okay, Alright⌠Hrm. This sucks.
Not necessarily. If you create a method that returns your data, and poll it twice a second, it might still be more performant than using a subscription for data that changes so frequently, and youâll still get all the updates
Wow really?
So actually get this. On my localhost, I copied the DB from production, so now I have a BUNCH of data.
Great. I loaded it all in.
Thereâs NO activity occurring on the DDP, but yet I still have this client lag. Any ideas? 300 documents, ~20 keys each.
Maybe tomorrow Iâll create a demo app to test this stuff.
Well I think what Iâll do is create zone instances. A client really doesnât need to see 10,000 players on screen anyway.
This is what every MMO does. Maybe thereâs a reason for it, that Iâm not immune to. That is a LOT of data subscriptions!
So if I limit to like, 20-40 documents, should fix my problem.
In our case documents often didnât get fetched at once, this especially resulted in multiple and unnecessary renderings on start.
We fixed that with isReady on the subscription handler on the initial load together with Reactâs PureComponents (or write your own shouldComponentUpdate) to avoid reactive unnecessary rerenders. Hope this helps a bit.
Yeah thatâs what Iâve been discovering. Iâm currently testing out a theory very similar to this.
Have you already see this answer (of mine) on StackOverflow: http://stackoverflow.com/questions/12298640/meteors-subscription-and-sync-are-slow/19599027#19599027. In my experience, the only thing that slows things down are reactive re-renders, so you want to disable them (using a guard or similar) before loading a lot of data. In iron-router you would make sure to load the data first and only then render the page (see âwaitOnâ).
you can use react virtualized (if youâre using react):
1000 rows no probs: https://bvaughn.github.io/react-virtualized/#/components/List
Could you post a CPU profile (or PM me)? Itâs the easiest way to figure out whatâs going on.
Maybe itâs already proposed in one of the comments, but try to ask yourself the question whether you even need this much documents? It might become more performant when using less documents (by adding i.e. more nesting). In that case you might need a little more advanced observers on the client (only watching particular parts of a document instance), but you can keep all the current (reactive) functionality.
I havenât any performance benchmarks so whether it could be really faster is a pretty wild guess⌠Whether itâs worth the refactoring of course also depends on the current structure and size of the application.
