There is already a collection named [...]

help
1

Mongo collections are not local in the sense that if you have in two different files the line

const myCollection = new Mongo.Collection('myCollection')

Meteor complains saying

There is already a collection named "myCollection" at new Mongo.Collection (collection.js:226)

To circumvent the problem I use a collection dictionary

const collections = {}

get_collection = name => {
    if (!collections[name]) collections[name] = new Mongo.Collection(name)
    return collections[name]
}

const myCollection = get_collection('myCollection')

But I am worried this might prevent the garbage collection of the wrapping object once the collection is not needed any more. This wasn’t much of a concert with monolithic applications but now that Meteor 1.5 allows code splitting, it might be the right time to solve this problem in my application(s)

What is the recommended way of proceeding ? (also, why creating collections in two separate files raises an error when const variables are supposed to be scoped per file / function) ?

2

Collection in Meteor should be declared once and you can access it anywhere in the server and in the client when published.

3

That is not the case.

You might need a collection only on the client or only on the server. There is no obligation to have it on both sides at the same time

  • You can manipulate collections on the server without publishing them
  • You can manipulate local collections in mini-mongo that are not linked to any table on the server.

And you may have different files / modules that use the same collection (= physical table in Mongo). In which case you may want to create a wrapper for that collection in each of your modules / files to keep them independent.

Also I have the impression you are somehow confusing the table in Mongo with the wrapper object in Meteor here.

4

I always avoid these problems by proper planning. This means including the creation of a model for my datastructure. By modeling i prevent naming collitions. If they happen during the modeling phase, often its a sign of bad naming. Its not exicit enough.

Locally scoped stores in my case are just reactive arrays or objects. If they do require minimongo I would namespace them with thw package name: {package}_{collectionName}

1 Like
5

To create Client only local and temporary collection, use:

const myColl = new Mongo.Collection(null);

Form the docs:

If null, creates an unmanaged (unsynchronized) local collection

Server only collections created at Server scope either /server directory or in a if (Meteor.isServer) { /*...*/ } block, same can be applied for synced Client collection. If collection is synced with Server we can not call it “Client only”.

Not sure if it’s answers your question, more info required to help you. What actually you’re trying to accomplish?

6

Create an empty meteor application with

app/server/file1.js

const myCollection = new Mongo.Collection('myTable')
export const d1 = myCollection.findOne()

app.server/file2.js

const myOtherCollection = new Mongo.Collection('myTable')
export const d2 = myOtherCollection.findOne()

Gives

(STDERR) Error: There is already a collection named "myTable"
(STDERR)     at new Mongo.Collection (packages/mongo/collection.js:240:15)
7

This is expected behaviour. You should create the collection only once.

// app/server/file1.js

const myCollection = new Mongo.Collection('myTable');
export const d1 = myCollection.findOne();

// app.server/file2.js

import myCollection from `/server/file1.js`;
const myOtherCollection = new Mongo.Collection('myOtherTable');
export const d2 = myOtherCollection.findOne();

Why would you have two collections with same name? You aren’t explaining the purpose of this.

8

Because you have two modules that use the same table in Mongo like “show data as a table” and “show data as a chart”

If you create the wrappers only once then you need a file containing all your collections from which you import the one you need for the specific module, but now your module depends on the existence of an external file

/module1/file.js

import { myCollection } from 'all_collections.js'
console.log (myCollection.findOne())

/module2/file.js

import { myCollection } from 'all_collections.js'
console.log (myCollection.findOne())

Or else you use a global dictionnary to ensure wrapper unicity which is what I do in my code. In that case your code only depends on the Mongo collection name (which I believe is reasonable).

/lib/collection_manager.js

const collections = {}

export const get_collection = name => {
    if (!collections[name]) collections[name] = new Mongo.Collections(name)
    return collections[name]
}

/module1/file.js

import { get_collection } from 'lib/collection_manager.js'
console.log (get_collection('myTable').findOne())

/module2/file.js

import { get_collection } from 'lib/collection_manager.js'
console.log (get_collection('myTable').findOne())

The only problem is the garbage collector may not clean the wrapper even if not used anymore because of the pointer from collections in /lib/collection_manager.js. What would be ideal there are ‘weak references’ (https://en.wikipedia.org/wiki/Weak_reference)

In computer programming, a weak reference is a reference that does not protect the referenced object from collection by a garbage collector, unlike a strong reference.

9

Well, let both libs to accept Collection Instance as a some kind of initialization argument.

10

How do I do that ?

I have

/imports/chart/chart.js

import { get_collection } from '/lib/collection_manager.js'
export const chart = name => get_collection(name).findOne()

/imports/table/table.js

import { get_collection } from '/lib/collection_manager.js'
export const table = name => get_collection(name).findOne()

/main.js

import { chart } from '/imports/chart/chart.js'
import { table } from '/imports/table/table.js'

// Some kind of router
if (location.href = "http://app/table") table('myTable')
else if (location.href = "http://app/chart") chart('myTable')

How do you transform that into a code that loads the module when the router directs to the proper page and cleans the collection wrapper once done ?

11

Something like:

///imports/chart/chart.js
export const chart = (collection) => collection.findOne();

///imports/table/table.js
export const table = (collection) => collection.findOne();

///main.js
import { chart } from '/imports/chart/chart.js';
import { table } from '/imports/table/table.js';

// Some kind of router
import { collection } from '/lib/collection_manager.js';
if (location.href = "http://app/table") table(collection);
else if (location.href = "http://app/chart") chart(collection);
12

I fail to understand how that solves the problem.

Imagine I have 10 000 collections and I don’t open any page that needs them.

import { collection } from '/lib/collection_manager.js

Then this line just allocated memory for nothing.

What you want is to:

  • open a page
  • create the corresponding collection
  • use it
  • clean as you open another page that doesn’t use that collection

That’s the reason Mongo.Collection is annoying because it doesn’t behave like a local variable.

13
  1. Will this collection be local and temporary or it will write/read data to/from Server?
  2. Why you believe GC won’t collect code of unused collections
  3. Why you care so much about GC on the Client? Will user visit all 10 000 pages to load those 10 000 collections? I believe average user won’t go further than 50 pages in a raw without page reload/revisit.

Again idk your use case, if you may explain a bit more about what are you trying to solve/accomplish this will let me help you. I’m asking to tell more only because it feels like you’re using a wrong tool, or a using it in a wrong way.

14

I agree with the others, I think your use case might warrant the use of something else instead, so with a little more info we might come up with a better solution.

By the way, if you have 10.000 collections in your db you have other problems to worry about :wink:

1 Like
15

The GC cannot clean this collection

/main.js

const collection = new Mongo.Collection ('myTable')

// Some kind of router
if (location.href = "http://app/table") console.log(collection.count())
else if (location.href = "http://app/hello") console.log("Hello")

The GC doesn’t know what page you will open. The GC cannot guess you will only open the page “hello” and the collection can be disposed of (actually didn’t even need to be created in the first place). That collection can only be cleaned when the application exits.

Imagine a Meteor application that is a MongoDB GUI.
There are 100 tables in the Mongo database
The application has two pages : read-only and edit.
The routes on the client take the table name as a parameter

When you enter a given route, the client

  • subscribes to collection
  • sends it to the proper widget for display

1. Why do you want to pre-allocate 100 collections when you don’t know if the user will view them or not ?

Now imagine the MongoDB table is accessed by other applications that may add or delete tables.

2. How are you going to pre-allocate the collections now ?

Let’s imagine you solve problems 1 and 2 (for instace using my get_collection function). Now the user opens and closes every day a different table without ever closing the browser.

3. How are you going to free the collections that are not used anymore ?

16

Because in almost all use-cases the resources this will take up is negligible. You’re probably over-optimizing if you’re worrying about GC in this case.

I can’t really come up with any use case where you would have 100+ collections. As mentioned before, if you have that many collections you’re likely doing something strange/inefficient from an architectural standpoint and you might need to rethink your data structures.

For the sake of experimentation, if you really want to push the boundaries with the amount of collections, this is what I’d advise:

  1. Do a few benchmarks to see the impact of creating 100/1000/10000 collections on the client, that way you’ll know if this is something you need to worry about at all. (feel free to share your findings, I’d be interested in the results)
  2. Delve into the meteor core code to see if there are any hidden methods to remove collections or similar.
  3. Extend your library approach to allow setting the variable containing a collection to null and do the necessary cleanup. (as in write your own removeCollection function). When finished you can possible make a PR for this in the meteor repository and have it implemented in meteor itself.
1 Like
17

There are many reasonable reasons not to pre-allocate tables, regardless of their number and the memory consumption.

I already gave an example where you CANNOT pre-allocate because the application doesn’t control the table creation - and a database GUI is not a obscure example of application.

Theare are many other reasons ranging from practicality to legal

You may have one table per customer (and say 10 customers) and you don’t want to merge them

  • For practical reasons (slightly different formats)
  • Legal reasons (your contract says the guy that cleans the data for customer A cannot be the same as the one that cleans the data for its direct competitor B)
    At the same time you want your tables in the database to have clear names like “CLIENT_clientname” and you don’t want all your customer names to be hardcoded in your code bundle.

You operate in multiple countries and each country has different laws regarding the information you can collect about your customers. It’s very reasonable to generate a table per country as new customers subscribe. Rather than 196 tables just in case someone from a remote country uses your service.

You operate in multiple countries and your data is in different languages.

[Add your example here]

In any case the idea that preallocating tables is reasonable for any application doesn’t hold. Also, your database organization ideally shouldn’t be dictated by the framework used to visualize the results.

18

I don’t agree that you should be creating unique collections for customers, instead you could structure your collection so you have a customer _id and perhaps a general data field that can have it’s own schema per customer. This works for a myriad of use-cases including the stuff you mentioned (per customer, per language, etc.).

There are no legal issues that can crop up if you limit access to the collection given a customer ID since you’re still hosting your data in the same database so there is no real isolation from data (my opinion). If you were using a different db for each customer that would be something entirely different.I’m no legal expert of course.

Either way, points [2] and [3] still hold if you still want to approach it the way you’re thinking. So creating client-side collections dynamically and then removing them when they become unnecessary. (by writing your own logic for the removal)

There are also possible workarounds depending on if the data has to be reactive or not:

  • If not reactive you can create your own collection (a simple array) on the client-side and call a method which would return the data.

  • If reactive then the solution would be slightly more complex, for example by creating a single general publication that accepts the collection name as a parameter. Depending on the complexity of the query you could use something as simple as https://atmospherejs.com/dburles/mongo-collection-instances or use the lower-level methods like .observe, .added, .changed and .removed.
    You could even write your own collection logic for the client-side (I did so with a react native project, using mobx and a direct ddp connection, but there might be a way to capture the ddp messages in meteor itself).

Safe to say your use-case is not that common and will require some work to implement, but there definitely are ways to do it, just not built-in as far as I know.

19

I have somewhat similar concerns to @diegoolivier.
I think there should be a way to de-initialize collections so that they can be garbage collected.

My use-case is different to @diegoolivier’s though.
What @diegoolivier is doing in terms of multiple distinct variables creating distinct collection objects, but with a shared/identical collection name, seems strange and one might argue there are problems with that approach… and it’s definitely not something that I would do personally.

This is what I’m doing:
I’m building an application where collections are generated dynamically, there’s no limit to the number of collections, but the collections that need to be accessed by the server/client depends on who is logged in and what they’re doing.

So instead of instantiating “a million” collection objects at startup, I’ll instantiate the specific collection objects that are needed, when they are needed.
However, it would be my preference to destroy the collection objects with a timer when they’ve not been used for a long time (an hour, a day, etc).

But currently there is no obvious way to do that, and so collection objects that are no longer needed, can never get garbage collected.

So for example if I do this:

allCols = {};
let collectionThatWeNeedRightNow;

collectionThatWeNeedRightNow = 'foo'; // this is dynamic
allCols[collectionThatWeNeedRightNow] = new Mongo.Collection(collectionThatWeNeedRightNow);

// some time later, kill it so the garbage collector can free memory
delete allCols[collectionThatWeNeedRightNow];

// okay we need it again
collectionThatWeNeedRightNow = 'foo'; // this is dynamic, we happen to need the same collection again
allCols[collectionThatWeNeedRightNow] = new Mongo.Collection(collectionThatWeNeedRightNow);
Error: There is already a collection named "profile"
  at new Collection (packages/mongo/collection.js:122:15)

So as packages/mongo/collection.js works right now, because there is no known way to delete collection objects from memory. What will eventually happen is my node.js instance will be unable to garbage collect, collections will be “opened” as needed, but never “closed”, then eventually node.js will get killed by the kernel (OOM out of memory) or self-kill based on node.js memory limits and get restarted by the process manager.

I think there’s no justifiable reason that there shouldn’t be a way to destroy a collection object when it’s not needed.