How to implement a dynamic collection?

Hello everyone, for a specific need, I would to create my collection dynamically. I would for every company has her own CLIENT collection :

var companyID = Session.get(“UserLogged”).code;
var collection = companyID+’_clients’;
Clients = new Mongo.Collection(collection);
var client = {
“name”: “XXXXXX”,
“code”: 122233
}
Clients.insert(client);

I got in my console insert failed: Method ‘/Lux-PEGL92iCaNXsk_clients/insert’ not found

I will appreciate any help :slight_smile:

you cannot create collections dynamically on server. If its only on the CLIENT (your example is confusing, because you use also the word client in another context i think), you have to pass null to the collection to create a CLIENT-only collection:

clientOnlyInMemoryCollection = new Meteor.Collection(null);

they live only in memory, if the CLIENT is terminated, these data is away.

if you need multiple of them, just use them like any javascript object, e.g. create some kind of map if you want to associate a collection to a companyId:

const companyCollections = {};

const getCollectionForCompanyId = companyId => {
   if(!companyCollections[companyId]) {
     companyCollections[companyId] = new Meteor.Collection(null);
   }
   return companyCollections[companyId];
}

1 Like

My collection name is Clients.

??

You definitely can create collections dynamically on the server. I think the issue @kiko91 was seeing was due to the fact that he was creating it only on the client. As long as the new Mongo.Collection on the server is run before the client tries using it, it will work just fine. So, typically, use Meteor method to create the collection on the server before trying to define it on the client.

Having said all that, I’m not convinced that the extra complexity of this approach is offset by the separation of customer data into multiple collections. A single collection may be efficiently and securely “split up” for mutliple customers - and makes the implementation and code much easier to reason about.

4 Likes

ok, propably it works, but “i’ve got a bad feeling about this” :wink: I am not sure, if in certain edge cases this would lead to problems.

I also would advice to keep it simple to begin with and just creating one collection for all clients, using a companyId to separate them (don’t forget the index), unless you have a good reason and experience why you would want to separate them

1 Like

You mean it’s impossible ? :disappointed:

No - you can do it if you want to. I just don’t think it’s a good way to handle multi-tenancy.

2 Likes

Actually I added the collection manually in the server (of course for now). My approach is the new Meteor.Collection(X); X would it be dynamic (depends from the information of who connect to the app). You understand what I want nah ?

Yes.

1 Like

You could also just give each company their own meteor instance. Then everything will be neatly separated, and give you a scaling benefit too. If there are some collections that should be shared between companies, you can put them in a separate shared db like this:

if(Meteor.isServer){
	Shared = new MongoInternals.RemoteCollectionDriver("mongodb://localhost:27017/shared?replicaSet=rs0",{oplogUrl:"mongodb://localhost:27017/local?authSource=shared"});
} else {
	Shared = null;
}
Messages = new Mongo.Collection('messages', { _driver:Shared });
4 Likes

@herteby’s suggestion is interesting, but creating mongo collections dynamically is a really really bad idea for many reasons, so you should have an equally good reason to do it.

Without knowing everything that you’re trying to do, here’s my suggestions. Since all you really need to separate them is the company._id, just add a field to the collection called ‘companyId’, then control data through subscriptions and find queries. For example:

// server side
Clients = new Mongo.Collection('clients');
Meteor.publish('clientsByCompany', function(companyId) {
  return Clients.find({companyId: companyId});
});

// and on the client side
var companyId = Session.get("UserLogged").code;
Meteor.subscribe('clientsByCompany', companyId);
Clients.insert({companyId: companyId, ...etc});
Clients.find({companyId: companyId});
2 Likes

Thank you guys. [quote=“buishi, post:11, topic:36219”]
Without knowing everything that you’re trying to do, here’s my suggestions. Since all you really need to separate them is the company._id, just add a field to the collection called ‘companyId’, then control data through subscriptions and find queries. For example:
[/quote]
That’s what I did :wink:

2 Likes

Hello everyone,
I’m digging this out because I am trying to achieve a similar thing for a different use case. I would love to have opinions/advice from more experienced Meteor and MongoDB developers such as @robfallows. Maybe someone has already tried a similar thing.

Use Case:
Storing huge (~ 1M rows with ~5 columns each) amounts of tabular data. (I know MongoDB is not the best bet for this, but I am curious if it could work, because I’d love to use Meteor for the project.)
My idea was to have dynamic collections for each table that I am storing and a “meta-collection” to keep track of those prefixed table collections.

Example (collections depicted as arrays):

tables: [
 {_id: radomId, name: tableprefix_tableA, customer: x, otherMetaInfo: ... },
 ...
],
tableprefix_tableA: [
 {_id: 0, c: [col_content1,col_content2,col_content3,col_content4,col_content5]},
 {_id: 1, c: [col_content1,col_content2,col_content3,col_content4,col_content5]}
 ...
]

Reasoning:
Why different collections at all? My assumption is, that if I throw everything into one collection, it will be a lot harder for MongoDB to query based on a “type/name” property and performance would degenerate with each table that I add to it.
Example query: {$and: [{name: tableprefix_tableA},{col.0: {$gt: 20} }]}
There is also the notion of “wasted” space, if I have to repeat the type/name-property a million times for each record (/row in that case).
Why one document per row? I thought that it would be the best way to store tabular data. If I use the unique rowNumber as the _id, it saves some space and it would be easy to get back a range to display to the user. More advanced queries could also be constructed fairly easy on the whole collection or certain ranges. It would also be quite easy to create an index on the columns of each table. I just have a gut feeling that if I store documents like this:

{
   tableName: 'A',
   rows: [
           [col1,col2,col3,col4,col5],
           ... (999.999 more rows)
   ],
   otherMetaData: ...
}

it could be a lot harder for MongoDb to query and I guess there is no possibility to build an index on the columns of the row(?).

Let me know if I went into a wrong direction with my reasoning. Thanks in advance. :slight_smile:

Welcome to the forums :slightly_smiling_face:

That’s not particularly huge (depending on how big your columns are). However, you’re absolutely correct in that you do have to understand exactly what you want to do and how to map that into a data architecture best suited to Meteor. That does not have to be MongoDB - You can use any database (with some caveats) as long as there’s an NPM package for it (which is to say, practically everything).

So firstly, why use MongoDB?

  1. Great integration with Meteor (isomorphic API).
  2. Tghtly coupled to the awesome Meteor accounts system.
  3. Great for unstructured data.
  4. Great for “big” data.
  5. Great for distributed data.
  6. When you need (easy) reactivity.

When wouldn’t you use MongoDB?

  1. When another database architecture would be a better fit.
  2. Where you don’t need reactivity.

This will be no harder/slower/less-performant than in SQL, for example. As with SQL, performance is hugely dependent on correct and well-formed indexes. If you haven’t analysed your queries (either by eye, or using .describe() in MongoDB) then you’ll be asking for trouble.

You will almost certainly not be able to do that in MongoDB, because there is a limit of 16MB per document. But then, you wouldn’t try to cram an entire table into one row of a SQL database, either!

MongoDB Indexes

Long story short: you can do this with MongoDB, MySQL, PostgreSQL, … in Meteor.

I’d still do it with a single collection though :smile:

2 Likes

Thanks for the warm welcome and sharing those insights. :slight_smile:

I didn’t realize that Meteor supports DB-Systems other than MongoDB out of the box. I didn’t read anything in the docs about it, but after your mention, I found some packages that add this functionality. This definitely opens up some doors and I will take a look into it.

Do you mean .explain()? This looks very promising. I am fairly new to MongoDB and never used .explain() before. But it seems like a great tool to use along with the profiler.

Is there any other reason other than added complexity (and I know that is a good one by itself already :smiley: ) why you wouldn’t use separate collections? To me it seems like it would be possible to save some space (for the additional index) and maybe achieve some minor performance gains, depending on the query.

Oops :flushed: - yes, you’re correct. My mind must have been somewhere else!