Thank you, I’ll try it out. I’m very excited about the results at the moment. Not excited about the price though. But I think I can get the replication set working myself.
WOW. This tool rocks! Way better than RoboMongo. My database is imported that quick! Fantastic.
OKAY, now lets see how the friggin app runs with OpLog finally enabled.
Finally, in summary, it appears the issue is resolved. OpLog.
Here’s a look at my new data load running my old database. It’s very fast now. Don’t have any users reconnected yet, but I’m none the less optimistic.
Btw, I think you could install Redis Oplog on Galaxy, instead of using Mongo’s Oplog. Would give you better performance too being on the same server as Meteor.
Also I googled “mlab oplog” and it says they have it:
http://docs.mlab.com/oplog/
Yeah their price is $740 a month! Meteor is suuuuper expensive. lol.
So still not happy, even with OpLog. Definite performance gains. But also, I wasn’t indexing ANY data. I have now properly indexed data.
My total db is 40,000 documents. Which makes sense earlier, why 40,000 documents were being created. If I have 2 users, that’s now 80,000 documents returned.
With indexing, back on my old non OpLog server, you can see the performance effects of indexing.
However, if we examine Kadira, you can see when I make the switch.
STILL bad.
Take a look at this. It may help you.
Answering the OP again (from an architectural perspective)…
I am not sure about Mongo but, generally, having an index doesn’t fix range queries (simply because ANY record could satisfy the query).
A lot of games of this nature usually have the concept of a “zone”. You would index “zone” and the user can only be in one “zone” so optimisation restricts the query set before it worries about the range query.
Once you have a “zone” you could use an idea similar to Google Maps (or Minecraft) and deliver the data for the “zone” only when the players movements require it. You can then use a client-side collection (that gets the progressive updates) and query that on the client side reactively.
Also, ensure that your data sets are “sparse” and not recording an “empty” cell if there is nothing in it (as Excel does when it stores data).
In any case, if I were to code a (potentially) MMO game in Meteor I would approach it that way.
In summary:
- Use Zones to segment the data into chunks (index on Zone and anything else that can limit the segment)
- Sparse data sets to reduce what is meaningful
- Progressively load new Zones into a client-side collection and query that
You can also use this concept on the server (with observers) to generate new zones as players start moving in certain directions.
If done right, this can provide a linear response time (or even close to static) versus an exponential one when the usage grows.
[UPDATE: Oh, and you could consider using a background job (vsivsi job collection is cool as you could even run a separate server to do the generation if required) to generate the Zones - one per job pass if there are pending Zones to generate.]
2 Likes
What are you referring to here? We use mLab shared, with opLog and it’s like $30/month for 2gb of storage.
Actually I was already thinking about this architecture earlier today. I think this is the approach I’ll need to take.
So I know what’s going on, and it’s pretty obvious in hind sight
I’ve got a Meteor SetInterval 1000 running on the server, which is calculating positions on all ships. Then every 60 seconds, it re-checks asteroids and planet behavior.
So, my database setup for OpLog is indeed efficient, but my CPU there is slow. I can’t mess with CPU settings for Mongo Atlas.
My Google cloud single MongoDB is certainly FAST but it’s super not efficient. One subscription was doing this:
4200 record searches. Now with Indexing that drops to:
A mere fraction.
So I think in this particular case of an MMO with rapid calculations in a loop, it’s pretty clear I’ll need super efficient and fast requirements.
If this were a normal application, probably not so bad. But then again, what if I did hit say, 500,000 users on a system. One would need this kind of information. Maybe it’s because I have no formal education, and this is simple stuff in an IT administration degree or something? Or maybe it’s just that complex.
Well. Here I go. Going to create my first cluster DB for mongo for oplog, and index it. That SHOULD be it.
I’ve built some amazing performant things over my lifetime on hardware that no-one would have ever dreamed of. So, I believe you could write a decent MMO using Meteor and Mongo as long as you get the architecture right and don’t try and build World of Warcraft (which most would agree is a job probably best written in C or C++).
FINALLY. IT. WORKS.
OpLog running on my own VPS with a baller CPU to do my stupid MMO 1s calculations.
Data is indexed.
OpLog is configured.
Performance is defiantly there!
Here’s the last screen shot.
I hope this massive research project to learning about Meteor + MongoDB Hosting is useful to someone in the future
In my opinion, it is poorly documented, and there is no warning when starting Meteor the massive undertaking it will be to host your production app. But, come visit me at www.StarCommanderOnline.com and see if I’m still online 
Please reply to this thread or contact me directly if you wish to ask questions.
THANK YOU to all who participated in throwing some knowledge at me. This community is something special. <3
I think, for most cases, Meteor+Mongo is fairly straight forward. An MMO really pushes the boundaries and I have a separate sideline project (not MMO but fan site related) that I am also building in Meteor that will need some social capabilities (where I hope to have thousands of users registered).
So, I’ve been thinking about the problems a bit but have not had a lot of time to write the code (due to the “paying” work for a finance related site). Just a parting thought… you may also end up needing to consider rolling your own commands across DDP (if the pub/sub is just too heavy).
Glad to hear that you are now having some joy with the stack.
1 Like
About the setInterval 1000 thing: For things like that, use the Collection.observe function. It basically accomplishes the same thing but without constant database checks 
1 Like
I will definatley check that out.
But the SetInterval does x+1, y+1 logic. Will this work? I’ll read up on it.