Hello everyone,
I’m digging this out because I am trying to achieve a similar thing for a different use case. I would love to have opinions/advice from more experienced Meteor and MongoDB developers such as @robfallows. Maybe someone has already tried a similar thing.
Use Case:
Storing huge (~ 1M rows with ~5 columns each) amounts of tabular data. (I know MongoDB is not the best bet for this, but I am curious if it could work, because I’d love to use Meteor for the project.)
My idea was to have dynamic collections for each table that I am storing and a “meta-collection” to keep track of those prefixed table collections.
Example (collections depicted as arrays):
tables: [
{_id: radomId, name: tableprefix_tableA, customer: x, otherMetaInfo: ... },
...
],
tableprefix_tableA: [
{_id: 0, c: [col_content1,col_content2,col_content3,col_content4,col_content5]},
{_id: 1, c: [col_content1,col_content2,col_content3,col_content4,col_content5]}
...
]
Reasoning:
Why different collections at all? My assumption is, that if I throw everything into one collection, it will be a lot harder for MongoDB to query based on a “type/name” property and performance would degenerate with each table that I add to it.
Example query: {$and: [{name: tableprefix_tableA},{col.0: {$gt: 20} }]}
There is also the notion of “wasted” space, if I have to repeat the type/name-property a million times for each record (/row in that case).
Why one document per row? I thought that it would be the best way to store tabular data. If I use the unique rowNumber as the _id, it saves some space and it would be easy to get back a range to display to the user. More advanced queries could also be constructed fairly easy on the whole collection or certain ranges. It would also be quite easy to create an index on the columns of each table. I just have a gut feeling that if I store documents like this:
{
tableName: 'A',
rows: [
[col1,col2,col3,col4,col5],
... (999.999 more rows)
],
otherMetaData: ...
}
it could be a lot harder for MongoDb to query and I guess there is no possibility to build an index on the columns of the row(?).
Let me know if I went into a wrong direction with my reasoning. Thanks in advance.