Good write up @awatson1978.
I’d like to add - to avoid sharding mongoDB, you have the option to shard meteor at the application level.
Simply back each Meteor cluster with a separate mongo instance.
You’ll need a (somewhat) centralised Auth database, then just use DDP.connect to connect to the right server for that Company/User. (eg, that contains all their tickets/projects/whatever)
If you read carefully, I said you need a replica set for an oplog - I didn’t specify any requirement about the number of instances. It’s well known, and for any new reader should be known, that Meteor by default runs its bundled Mongo as a single instance replica set.
as for polling, processes may arbitrarily “skip” the delay, but it’s not a guarantee that the average (don’t mean to sound condescending) developer would expect when they read that “Meteor is reactive!”
You mean here:
There’s no arbitrary part here - If you are using a single meteor instance - all db interaction done via that instance is 100% reactive, even using the polling driver.
Only when writes go directly to mongo from other applications (or other meteor instances) there may be up to 10s delay with the polling driver.
I’ve further clarified my statement in a following post about the single instance terminology
as for polling, it is not exceptional for an app to grow beyond a single app instance and it’s just something that needs to be very clear to any Meteor developer from the start, since it’s an architectural factor to bear in mind
I think our discussion has concluded, as we’re both on the same page 
The important outcome is I hope fellow post readers can understand better the limitations of Meteor and Mongo
Thanks for all your time with this topic. I figured it’d take a lot before this became an issue but just wanted to make sure.
@awatson1978 Thanks for your comprehensive reply.
I realize that the concern about data-consistency is really a problem if my app is successful - in other words - a problem I would like to have. At the same time, I wouldn’t want to have to change the whole stack at that point.
I’m thinking about the possibility of ‘backing-up’ the crucial data to PostgreSQL, then syncing the MongoDB server with it in the event of a system crash. This would involve dual writes to both Mongo and Postgres.
I’m also intrigued to hear about TokuMX. Their enterprise pricing seems quite affordable, provided that their storage engine lives up to its claims (there’s also the free version). Have you any experience with it?
In the meantime, I’ll continue building my app on Meteor, and cross bridges when I come to them.
I agree with this, these seem like very important things to think about - but if I can manage a consistent storage system with mongo at low scale, I think the best option is to go with something that works now - see if the app is even successful, and then tackle problems that arise at 10^9 documents later.
@awatson1978 what a wonderful write up - so much knowledge in here that I did not know. I didn’t even consider insuring faults by the server - this might actually be the road I take. This is a great thread, thanks @paryguy for starting this - I think some very good discussion came of it.
Glad I could help! Best of luck with your projects!
Will Mongo DB’s purchase of WiredTiger’s tech help with any of the issues people are having with Mongo?
I humbly suggest folks spend a little time looking at Meteor code, Mongo code, Javascript, JSON, and seeing why certain components were chosen. Trendy concerns of late on hackernews are as useless on the criticism side as they are on the bandwagon side (FUD)… Mongo seems like everyone’s favorite horse for beating as of late, and that Postgres manual sure is thick, I printed their docs once, about 15 years back, and that used a lot of trees.
SQL (you might as well mention Prolog) has nothing to do with anything I mentioned in my first sentence, but it’s a good thing in it’s own right, but I question heavily layered software that doesn’t firmly stack-up one-turtle-upon-the-next, as an ORM between SQL and Javascript objects would need to be… It’s not saying that it’s not hackable, it’s just not as elegant. Let’s put it this way: Mongo DB perfectly mates with Javascript on the server, and if I were interested in SQL, I would perhaps base my entire server side platform on the premise of a SQL connector, from the beginning.
Meteor provides a rather thin wrapper to Mongo calls, and actually, Mongo is pretty darn simple to get up to basic speed with!
Start up a Meteor app, and then run “meteor mongo” in another terminal tab and have some fun poking around “show collections” and then “db.whatever.find()” or getting into Mongo selectors and projections… super simple stuff actually…
Yep, as much fun as it is to jump on the Mongo is bad band wagon. They are aggressively expanding and things will only get better, for example the guy that founded Wired Tiger wrote the paper on snapshot serialization that PostgreSQL uses. I mean MySQL was basically in the same boat when it came out and people insulted it for its poor consistency guarantees, that is until InnoDB came along. Now there is a lot of stuff not related to the storage engine that needs to be fixed but I can guarantee you that the people at Mongo know about peoples concerns and working to fix them.
Thanks for the insight everyone. That’s why I originally posted the question, it just seemed like out of nowhere a bunch of MONGO IS BAD posts and articles started to show up. But I guess that comes with a growing developer community.
Sorry for necrobumping, but some of you might be interested to know that Mongo did have a bug that didn’t guarantee what the documentation promised but it is fixed as of Mongo 3.4, and Mongo 3.6 is looking very nice!
Meteor 1.5.1 is still on Mongo 3.2. Let’s hope an update comes soon!