Well looks like I’ve reached the end of the tunnel.
Meteor OpLog is the term you’ll want to google. It’s a beast. It’s what gives Meteor speed on production systems. And it’s SUPER hard to enable on your own custom deploy.
https://meteorhacks.com/mongodb-oplog-and-meteor/
Looks like I’m going to try mLab again and see if that don’t fix my problem…
MongoDB Atlas (by the Mongo people) supported OpLog out of the box.
Check this article by OK GROW for a comparison between the major Mongo services…
Yeah I think I’ll have no choice. I’m testing how to replicate servers at the moment, frigging nuts man. But hey, it’s the future; welcome to nuts? lol.
Thanks for the link.
Oh my lord, the pricing on MongoDB hosting is INSANE
Compared to mLab ?
[The forum wants me to write more than 20 characters]
Oh, you don’t have Oplog enabled? You HAVE to have it, otherwise Meteor will totally trash your database. I just host Meteor on our office server (regular old computer). Set it up myself with Mongo and Nginx 
The Meteor dev bundle actually has Mongo with Oplog built in btw.
https://www.okgrow.com/posts/mongodb-atlas-setup
This guide was very good, I have a database setup and running. Sadly I cannot import my old data easily… So I created some test data.
It’s much better.
Correct, I didn’t have OpLog running. Never heard of it. Haven’t seen any guides about it. Friggin crazy, but looks like it may solve the problem. I really need to import my data, but it’s giving me a friggin nightmare to import it.
Transferring data is super simple if you get Studio3T. You can just right click and copy-paste collections or even entire databases, with options to either replace or merge And if you don’t have direct access to Mongo it can tunnel through SSH.
Thank you, I’ll try it out. I’m very excited about the results at the moment. Not excited about the price though. But I think I can get the replication set working myself.
WOW. This tool rocks! Way better than RoboMongo. My database is imported that quick! Fantastic.
OKAY, now lets see how the friggin app runs with OpLog finally enabled.
Finally, in summary, it appears the issue is resolved. OpLog.
Here’s a look at my new data load running my old database. It’s very fast now. Don’t have any users reconnected yet, but I’m none the less optimistic.
Btw, I think you could install Redis Oplog on Galaxy, instead of using Mongo’s Oplog. Would give you better performance too being on the same server as Meteor.
Also I googled “mlab oplog” and it says they have it:
http://docs.mlab.com/oplog/
Yeah their price is $740 a month! Meteor is suuuuper expensive. lol.
So still not happy, even with OpLog. Definite performance gains. But also, I wasn’t indexing ANY data. I have now properly indexed data.
My total db is 40,000 documents. Which makes sense earlier, why 40,000 documents were being created. If I have 2 users, that’s now 80,000 documents returned.
With indexing, back on my old non OpLog server, you can see the performance effects of indexing.
However, if we examine Kadira, you can see when I make the switch.
STILL bad.
Take a look at this. It may help you.
Answering the OP again (from an architectural perspective)…
I am not sure about Mongo but, generally, having an index doesn’t fix range queries (simply because ANY record could satisfy the query).
A lot of games of this nature usually have the concept of a “zone”. You would index “zone” and the user can only be in one “zone” so optimisation restricts the query set before it worries about the range query.
Once you have a “zone” you could use an idea similar to Google Maps (or Minecraft) and deliver the data for the “zone” only when the players movements require it. You can then use a client-side collection (that gets the progressive updates) and query that on the client side reactively.
Also, ensure that your data sets are “sparse” and not recording an “empty” cell if there is nothing in it (as Excel does when it stores data).
In any case, if I were to code a (potentially) MMO game in Meteor I would approach it that way.
In summary:
You can also use this concept on the server (with observers) to generate new zones as players start moving in certain directions.
If done right, this can provide a linear response time (or even close to static) versus an exponential one when the usage grows.
[UPDATE: Oh, and you could consider using a background job (vsivsi job collection is cool as you could even run a separate server to do the generation if required) to generate the Zones - one per job pass if there are pending Zones to generate.]
What are you referring to here? We use mLab shared, with opLog and it’s like $30/month for 2gb of storage.
Actually I was already thinking about this architecture earlier today. I think this is the approach I’ll need to take.
So I know what’s going on, and it’s pretty obvious in hind sight
I’ve got a Meteor SetInterval 1000 running on the server, which is calculating positions on all ships. Then every 60 seconds, it re-checks asteroids and planet behavior.
So, my database setup for OpLog is indeed efficient, but my CPU there is slow. I can’t mess with CPU settings for Mongo Atlas.
My Google cloud single MongoDB is certainly FAST but it’s super not efficient. One subscription was doing this:
4200 record searches. Now with Indexing that drops to:
A mere fraction.
So I think in this particular case of an MMO with rapid calculations in a loop, it’s pretty clear I’ll need super efficient and fast requirements.
If this were a normal application, probably not so bad. But then again, what if I did hit say, 500,000 users on a system. One would need this kind of information. Maybe it’s because I have no formal education, and this is simple stuff in an IT administration degree or something? Or maybe it’s just that complex.
Well. Here I go. Going to create my first cluster DB for mongo for oplog, and index it. That SHOULD be it.
I’ve built some amazing performant things over my lifetime on hardware that no-one would have ever dreamed of. So, I believe you could write a decent MMO using Meteor and Mongo as long as you get the architecture right and don’t try and build World of Warcraft (which most would agree is a job probably best written in C or C++).
