Meteor with multiple Mongo's connection?

help
1

Hello,

I can see same topic but without responses… We want to use multiple mongo connection (at least 5) (permit user to see data for brand A or brand B, like Stripe for ex.). Workflow is :

  1. User choice Brand A
  2. All requests, call, publish, etc… pass trough mongo connection for Brand A
  3. User choice Brand B
  4. Same interfaces than before, but meteor pass trough mongo connection for Brand B

We make some test with RemoteCollectionDriver, but it’s hard to know exactly where to put this segmentation

Is someone already implemant somethings like that ?

2

One Meteor App can only have one Oplog so you can only publish from one replica set but you can use as many DBs you want, with methods. You just create multiple connections/collections as you wish.

// declare your DB uris
process.env.MONGO_BRAND_A
process.env.MONGO_BRAND_B // etc

const _driver = new MongoInternals.RemoteCollectionDriver(process.env.MONGO_BRAND_A, {})
const brandA = new Meteor.Collection('brandA', { _driver })
brandA.find().fetch()

// etc

// reactivity only for MONGO_URL when with a declared MONGO_OPLOG_URL or with Redis-Oplog:
3

Can’t we use oplog with RemoteCollectionDriver ? We use oplogToRedis yes

What seems to work better for now, is to declare a global dict on startup, and use it on methods / publish / etc.

But it’s hard to monitor real impact…

I see that it’s possible to passtrough other meteor’s instance to use it as a database layer. Someone experiment that here ?

4

You’re probably going to want to patch the Mongo.Collection class to have the _collection and _driver be getters. You then use a meteor environment variable to specify the band, then any db calls issued from within a fiber tied to a band would use the relevant db connection. This way your application code doesn’t need to care about different bands beyond selecting them. For the oplog. If the databases exist in different clusters you’d need to use something like redis oplog and use custom channels in publications and updates where you prefix the collection name with a band identifier.

There are other ways of doing it, particularly if you don’t need to care about the meteor.users collection (or other collections defined by third part packages) it will depend on your exact use case.

5

Thanks for your reply znewsham.

For now, on server part, we create a array for each brand’s collection (invoice[brand] = connection) and on publish or method, we send brand as parameter to use corresponding collection.

BRANDS.forEach((brand) => {
  const connectionUrl = BRAND_CONNECTION_URLS[brand.id]

  const connection = new MongoInternals.RemoteCollectionDriver(connectionUrl)

  const collection = new Mongo.Collection(collectionName, {
    idGeneration: 'MONGO',
    _suppressSameNameError: true,
    _driver: connection,
  })

  FavoritesCollection[brand.id] = collection
})

It works, but we’re unable to use redis oplog. When we enable debug, it never change pubsub channel (only connected to last brand on startup array). But I am afraid that this is unmanageable with respect to performance.

publishComposite('allMyFavorites', ({ brand }) => {
  return {
    find: () => {
      return FavoritesCollection[brand].find(
        { user: user._id },
        { sort: { createdAt: -1 } },
      )
    },
  }
})

Cursor returned is not link to brand choice. If we debug with

console.log(FavoritesCollection[brand].find(
        { user: user._id },
        { sort: { createdAt: -1 } },
      ).fetch())

Here it return good result depend of brand sended.

We try with COLLECTION.configureRedisOplog, without success.

We’ve a meteor database, wich countains account & different global parameters, and one database for each brands (wich contains exactly same collections / fields on)

If someone have idea…

6

NB: Strange, my reply was deleted…

Thanks for answer znewsham

On server part we create an array of brand’s collection

BRANDS.forEach((brand) => {
  const connectionUrl = BRAND_CONNECTION_URLS[brand.id]

  const connection = new MongoInternals.RemoteCollectionDriver(connectionUrl)

  const collection = new Mongo.Collection(collectionName, {
    idGeneration: 'MONGO',
    _suppressSameNameError: true,
    _driver: connection,
  })

  FavoritesCollection[brand.id] = collection
})

And on each subscription/methods, we send brand as parameter

publishComposite('allMyFavorites', ({ brand }) => {
  return {
    find: () => {
      return FavoritesCollection[brand].find(
        { user: user._id },
        { sort: { createdAt: -1 } },
      )
    },
  }
})

It works, except for redis oplog. It never change channel (aka database) and always connected to last brand on BRAND_CONNECTION_URLS

We try with COLLECTION.configureRedisOplog but it not works.

For now we use mongo oplog, but I am afraid that this is unmanageable with respect to performance.

We’ve a “meteor” database wich contains account and global settings, and one database for each brands (wich contains exact same collections / fields on)

If someone have idea

7

I’m not sure what you mean here, redis doesn’t connect to mongo. The redis-oplog package does intercept the writes, but it should do that at the collection level correctly. Do you mean it only tracks changes correctly for the first/last subscription? If you’re talking about reactivity all of your collections have the same name, so redis itself will propagate the changes to each server correctly, but redis-oplog might not be able to maintain the correct multiplexers if the same user requests favourites from multiple brands at the same time. From the browser, what is the symptom of what you’re describing?