Meteor with the new MongoDB 3.2.1 Compose.io (in 2016)

mcchrish · February 10, 2016, 2:48am

Hi Everyone,

Compose.io recently update their MongoDB services with a new architecture, added security and more. Pricier too.
https://www.compose.io/articles/composes-new-mongodb-and-what-you-need-to-know/

Has anyone able to make Meteor (my current version is 1.2) work with the new setup? It looks like Meteor is having trouble working with MongoDB 3.2.1 (github issue) which is the only version Compose.io offers with their new service.

I know Compose.io still offers their older service through “MongoDB Classic”. But it’s a bummer if we can’t get to utilize the more advance service that Compose offers.

tjmonsi · February 10, 2016, 3:43pm

It seems we need to wait for Meteor 1.3 to come out before we get Meteor 1.4, which would include Node v4, which would hopefully include a driver to allow us to use MongoDB 3.2.1

As for me, I’ll still use MongoDB Classic (for now)… but I am looking around the web on how to upgrade from MongoDB classic to the new MongoDB deployment (in Compose io), any luck on that?

marbemac · February 10, 2016, 3:56pm

This process is described on the compose blog: Compose's new MongoDB and what you need to know - Compose Articles

We recently experimented with this, but there are indeed blocking issues. Meteor cannot connect to the new compose deployment yet.

mcchrish · February 11, 2016, 5:41am

Not an actual migration/upgrade guide. Just a tutorial on how to set up their new MongoDB deployment.

Thanks guys for the replies

tjmonsi · February 11, 2016, 6:42am

Would that mean we can already use the new mongodb deployment with meteor
1.2/3 just by using 1 host uri?

mcchrish · February 11, 2016, 9:31am

I tried this on a local meteor instance by defining a custom MONGO_URL that points to the one compose.io host URI but wasn’t able to make it work. I’m getting the same error with the mongo drive.

pcm · February 11, 2016, 4:41pm

Been playing around with this all week, as I’m setting up for production on Galaxy with Compose. I’m still in the midst of deploying, but here’s what Ive discovered so far:

Using a local meteor instance, I can connect to Compose’s new Mongo 3.2 setup, but I can only use 1 node. Adding a 2nd node gives the error: “Cannot read property type master of undefined” which seems to be coming from the npm-mongo driver.
Don’t use special characters in your connection pwd. (doh!)
They’ve got a new setup for oplog tailing. It seems to be working okay. Just to note, it costs an extra $4.50 per month to enable.
Compose enabled SSL, but I can’t get it to connect… Not sure if this is another driver issue.
They offer 1 month free, so I’m going to try and do a side by side with their Mongo Classic setup.

Losing the failover node kinda sucks. I’m not sure if the benefits of Mongo 3.2 outweigh that… But sure would be great to get an updated driver. Now that Compose is making 3.2 their default option, as the default recommended vendor to play with Meteor and Galaxy, it seems like this would be high priority.

Anyways, I’ll report back on any more findings.

nearpoint · March 30, 2016, 3:29pm

I have been trying to set this up to.

Using Meteor 1.3 with the new Compose.io

I can connect to compose.io but if I add the ssl=true parameter to the URI it does not work.

Crashes with “Cannot read property type master of undefined”

So I have not been able to get SSL working with new compose.io and Meteor 1.3. But works without SSL.

Also oplog did not work for me.

voidale · July 21, 2016, 6:25pm

Anyone had any luck with this?

mjmasn · July 22, 2016, 9:43am

Yes, but only by turning off SSL validation, which sorta defeats the point of having SSL (because you will still be vulnerable to Man-In-The-Middle attacks). I had to use the new Mongo.setConnectionOptions function inside a package (not an import) and I had to make sure that package was above accounts-base in the packages file (and the same for any other package that creates a Mongo collection because the options need to be set before the connection is first initialised). This isn’t ideal seeing as we’re trying to move away from using packages.

Put this in the package:

Mongo.setConnectionOptions({
    mongos: {
        ssl: true,
        sslValidate: false
    }
});

and your environment variables need to look something like:

export MONGO_URL="mongodb://user:pass@....dblayer.com:10000/my-database"
export MONGO_OPLOG_URL="mongodb://user:pass@....dblayer.com:10001/local?authSource=admin"

Note that oplog access is a paid add-on for the new MongoDB, at $4.50 / month I think.

I think it would be helpful if a Meteor dev could write a working example using Compose’s MongoDB+ with oplog (actually now known as just MongoDB, old MongoDB is now known as Classic MongoDB). A lot of people seem to be using Compose with Meteor but are stuck on the old insecure ‘classic’ setup. Any thoughts @benjamn? @sashko?

abernix · July 22, 2016, 3:10pm

I think you’re pretty close!

Disclaimer: This all relies on you using Meteor 1.4 which has a new Mongo driver, currently available as 1.4-rc.1.

This shouldn’t be the case. Simply use:

import { Mongo } from 'meteor/mongo';
Mongo.setConnectionOptions({
  /* options here */
});

…early in your application initialization, before the first Mongo.Collection is loaded.

As far as the SSL validation (which yes, is important), you’ll need to make sure you have SSL enabled on your Compose MongoDB+ account then obtain the “SSL Public Key” using their interface and load that file into your Meteor project using the Assets API.

Essentially, put the contents of the “SSL Public Key” into private/mongo-ssl.pem then:

Mongo.setConnectionOptions({
  mongos: {
    ssl: true,
    sslCA: Assets.getText('mongo-ssl.pem'),
  },
});

I haven’t tested this, but it should work in theory. And yes, I think it should and can be easier. If this works, will you open a new issue?

(Note: Due to #6784, Assets is not importable from anywhere but can instead just be used wherever)

abernix · July 22, 2016, 3:20pm

Also, in regard to your comment on GitHub:

I’ve used multiple Mongo hosts recently. Shards are slightly different but the Mongo URL format is the same and should be provided to you by Compose.io in a format that should work. Essentially, multiple host:port's are comma delimited after the username:password@ but before the /.

Again, this is all dependent on the new Mongo driver available in Meteor 1.4 (now in rc.1). And there is currently one reported Mongo issue in rc.1 that could be relevant.

mjmasn · July 22, 2016, 5:37pm

I stand corrected on all but the package bit. Although I have only just updated to rc.1 from beta.9 so possibly things were fixed in the meantime. Multiple URLs are working now, and the other issue was my fault - I’m loading the PEM file from an environment variable and had broken some whitespace somewhere.

I tried putting the config without the import statement in /server/lib/a.js to get it to load first but I get the error MongoError: connection 0 to x.y.dblayer.com:ZZZZZ closed. With the import statement I just get

/Users/mike/.meteor/packages/meteor-tool/.1.4.0-rc.1.k1qgx1++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:313
 						throw(ex);
 						^
 (STDERR) [object Object]

This is fixed by moving the config back to the package. I’m pretty sure packages load before anything else in order for them to be available in my code. Therefore after accounts-base initialises the users collection it’s too late to set configuration options. Happy to be proven wrong though.

Thanks for the help @abernix

Side note: MongoDB Atlas has now launched. Not sure what performance is like but appears to be much better value than Compose once you get above 3GB with MMAP or for any WiredTiger deployment. You start with 10GB Storage / 2GB RAM with Atlas vs 1GB / 100MB with Compose though so performance should be better. Atlas is WiredTiger only so to compare with Compose’s WT offering it’s half the price and has twice as much initial storage and RAM.

abernix · July 22, 2016, 5:57pm

Actually, accounting for accounts-base, you’re very likely correct. Please open an issue for this!

a.com · December 24, 2016, 1:29am

Is it possible to add configuration to an existing project? Or would you need to create a package and add it when you’re initializing the project? What are defaults for connectTimeoutMS and socketTimeoutMS? I have an large aggregation that keeps timing out with MLab no matter what I’m doing.

abernix · December 27, 2016, 10:49am

You probably want to read about how connectTimeoutMS and socketTimeoutMS work – I’m fairly sure you’re also setting them in the wrong place entirely wrong. They affect the connections to Mongo but not the operations and therefore they should be set on the connection string (i.e. MONGO_URL). They will not solve an issue with a large aggregation operation timing out.

See the connection string documentation for more about their defaults. Also, this is a good link to read to learn about “failures”:

https://mongodb.github.io/node-mongodb-native/2.0/tutorials/connection_failures/