Including my users collection, I have a fairly large app that has 5 collections.
I’m building a totally new part of our app, which might need an additional 2 collections, so I’m starting to feel like it is becoming a lot.
How many collections do you have in your app, and if so, were there any challenges with it?
We worked on nearly two dozen prototypes that had 5 to 7 collections. That seems to be a very typical number for Meteor apps, which usually includes user accounts and roles, maybe a statistics or cronjob collection, a master collection, and one or two data dictionaries.
After a lot of experimenting, we’ve managed to do a production refactor, and start pulling in all of our prototypes into a single master application. Nowdays we’re up to 25 or 30 collections, I think. And our design eventually calls for nearly 100 collections.
Challenges…
a) data modeling - To have a multi-collection app, you need clear business processes. Do business workflow analysis, do UML diagrams, figure out process state, cross reference with incoming/outgoing datatypes, create simple-schemas, and try not to reinvent the wheel.
b) pipelining - Collection hooks are your friends. Particularly with inbound/outbound message queues.
c) statistics - You can’t fit all the data from large-collection apps into minimongo. So, to coordinate them all, you need a statistics collection. Start with a singleton pattern and then move to a sorted snapshot pattern; use the most recent available stats record as a bucket to collect collection counts.
d) cron - You’ll need some regular maintenance; whether it’s just collecting statistics or more complex housekeeping.
e) dashboard / views - Ostensibly the problem that GraphQL tries to solve, a great pattern is to store your view state as a record in a collection. You have to identify all of your joins and lookup logic, and move it to to an insert(), update(), before.insert(), or before.update() function.
f) collection topology - As the number of collections grows, you’ll starting experimenting with indexing them, capping, sharding, write concerns, etc. Then you’ll start having decisions about how often capped collections should be flushed; whether users can control that; what your shard indexes should be; etc. etc.
$0.02
One app with 24 collections, one with 26. And I’ll be adding more collections to these apps, as the need arises. I’ve been careful to keep a lid on the amount of data sitting in minimongo and haven’t encountered any problems in that regard, so far.
I have multiple apps that have over 50 collections.
As long as yo structure your app properly, use good naming, and namespacing if applicable, it’s really not that hard.
71 collections
Our devs made some bad naming choices, but you just have to understand your business and remember what goes where!
incase you’re curious
Hah, this is awesome, thanks for sharing your experiences everyone! This makes me feel much better about our efforts!
@maxhodges Great list, Actually seeing all that is needed for a large scale app is fantastic, I can see that you’re basically using collections for everything you can, probable even very small but important needs (I like the shipping rates example).
As a follow-up question, I’m currently using Compose for my DBs, and at $31/Gb, I’m wondering how that scaled for you, do you have an estimate of the size of your DB and whom you are hosting with? Aren’t you quickly spending insane amounts of money to store all of those things ?
@awatson1978 Super interesting stuff. I’m nowhere near what you’re doing with your collections. Can you elaborate on your statistics collection, what does it do (is it a daily CRON that tracks inserts/updates etc.)? I don’t understand what joins and lookup logic are heh!
In general you have awesome insights into more advanced topics not often covered by most guides/tutorials aimed at getting a tiny app off the ground. Instead of trying to explain, maybe you have a couple of useful links to books (or anything else) that could help out when building your first (or not!) large scale production app?
It’s really not that large of an app in terms of data. We are 4GB on disk about 3 years since launch. RAM is the real bottleneck so we are paying $170 monthly at Compose for 7GB of disk to get 715 MB of RAM.
We also hosted on mLab and Atlas. mLab was the most trouble-free and has the most to offer in my experience. Will likely return to them at some point.
I have two collections.
I’ve added a type field to all my documents. So I can just subscribe by type.
I followed the Wordpress methodology of database objects, but ended up simplifying that even.
