Issues we had moving our MongoDb to MongoDb Atlas from Compose.io

deployment
1

I’d like to share my experience so far with Atlas and MongoDb.

Based on comments I’ve read on this forum and the blog post on OKGrow! I’ve decided to give Atlas a go. The second reason was more economically, as Compose.io (where my db has been working without a single problem for almost a year now) is charging unfortunately in their Classic plan (which isn’t Wired Tiger) for empty databases (each collection is a file and they keep the maximum size for that file, even when you delete all docs from it). That put me on 4 GB plan for a database of less than 32 Mb (!!!) in size.

Switching over to Atlas is hard, they really make it tough (I gave them the feedback several times). Why?

Because they really don’t offer an easy way to import your data from a competitor. Most of their options require you to be a command line wizard and a Linux expert (I’m neither as you can guess).

Why is it so difficult to have an import functionality for a backup from Compose.io (which is in that .ns file format)? The post from OKGrow! didn’t help either, found a solution on SO but again didn’t work. Reason, using different versions of MongoDb (the one on Compose is a rather old one).

So after upgrading via homebrew and with the help of my employee we’ve got the data up at Atlas.

Everything seemed fine, for about an hour. I’ve noticed the problems, I couldn’t connect. Turns out, in the free version (which I’m trying out as I’m not fully convinced that Atlas is the better choice) you can’t query the Oplog.

I spare you the details but check out this lengthy thread on MongoDb’s JIRA with their support to get your own opinion: Can’t connect and seems primary is down - No primary detected for set Cluster0-shard-0

In the last two days things got much worse. Not that I pointed my Meteor apps to Atlas, it’s been down twice. Currently it’s down, as I can’t log into my app. I can’t connect via their frontend either:

1308×476 56.5 KB

Same for the command line and for MongoBooster or any other tool I’ve tried.

The cluster is down. See another support ticket on JIRA for the cluster not being available: Production cluster is down

Luckily we are still in beta but with no one being able to log into the production system the beta testing has come to a stop.

So I guess you probably understand why I put up the provocative headline. Support so far? Well, apart from messages in JIRA (which is complicated to read as messages aren’t posted in a chronological order) they don’t feel the need to take any action. It’s weekend anyway and who cares about a potential customer’s free database that is down?

Guess what?

No one at MongoDb

So all in one it’s a complete disaster IMO, I will give them a chance to respond once it’s Monday in the US but it will be very difficult to convince me that this is a once in a lifetime rare occasion use case.

Unfortunately WT at Compose.io is pretty expensive and only makes sense when we go into production which is hopefully soon.

Andreas West

Co-Founder and CEO, Numero

Visit us at https://www.dnagenealogy.tools

1 Like
2

Nice post, summarizes the problems with mongoDB as SaaS
Still one major point is missing and is network latency between your application server(s) and mongoDB servers
My one cent of advice:
Get 1 small vm instances (3 if you want replication) in same datacenter where your application server(s) reside and install mongoDB yourself

  1. Its not a big deal installation wouldn’t take you more than 1 hour
  2. MongoDB is pretty stable once you set it right it takes care of itself doesn’t require any maintenance except upgrading versions from time to time which can be done without any disruption
  3. It will cost you peanuts ~$5 per server/month and you can upgrade the vm if needed without a disruption if needs arise.
  4. Your application communicates with mongoDB over the high speed network and fast DNS resolver of the datacenter.
  5. It will be pretty secure because you will not have to expose it to the internet but rather use your own private network.

Regards
Nick

5 Likes
3

Added a small but important detail.

1 Like
4

I’ll share my experience that I found their service easy to use, and really good.

But also expensive.

I’m self hosting now.

5

I would lean of Atlas over Compose because they use dedicated resources for your database and enable Write Concerns. However, it looks like they are still working to set things up.

1 Like
6

backups are not really an issue in self hosting mongo

  1. you can set a delayed replica set member in a different data center and this will protect you from a disaster.
  2. you can ran automatic cron jobs for mongo dump or disk snapshots
7

What do you mean by “they use dedicated resources for your database” and “enable write concerns”?

8

Any guides for setting up your own Clustered Mongodb instance?

I tied the free tier of Atlas but didn’t get far because of the lack of access to the op-log_tailing. I was about to test out their paid tier but will hold off based on your experience.

9

Number 1 is NOT a backup. That’s the same kind of thinking which goes “RAID 1” is a backup.

It is NOT.

Because if you accidentally delete/change data from the DB then that deletion/change will be propagated through the replica sets.

Also, follow the 3-2-1 rule for backups: At least three(3) different backups on at least two(2) different media with at least one(1) backup residing at a completely different location altogether.

Lastly. test your backups.

1 Like
10

Please read again I was talking about delayed replica member any accidental delete/change will NOT propagate but will be delayed by the set number of time interval.
And yes I agree on multiple backups policy that’s why I included #2.

11

you can start from official installation guide
besides

  • many cloud providers support a single click mongodb installation option
  • You can find plenty of containerized installations
    Having sayed that I very much prefer manual installations because it is more easy to configure options according to your particular use case, which you also cannot access if you use mongoDB SaaS.
12

Is it not possible to just connect to Atlas with Studio 3T and import a database dump from Compose? Thats what I was planning to do.

13

That’s nice. You set the interval to 30 seconds and notice the error after one hour.

How much help was the delay in that case?

14

Only a stupid would set the interval to 30 seconds,
I have many installations we set the interval to x hours, enough time for db integrity checks we run in the background to alert something went wrong.

15

No, because the database dump from Compose is in a different format (.ns files). Like I wrote the main challenge why the solution posted on SO didn’t work was that Compose.io is using a much older MongoDb version than MongoDb Atlas which is using right now the most recent (3.4.4).

16

Somehow my original thread got hijacked into a “How to run MongoDb in a self hosted environment” thread.

It probably would be better to open a separate thread for the self hosted part and stick to the original topic of sharing experience when moving from company xx to MongoDb Atlas.

Thanks for your understanding!

17

It’s not about db integrity. It’s about user error.

18

Okay, I always use Studio 3T to export my database also. If you try that it should be easy to import it into Atlas. No need for .ns files I think.

19

We just moved over to Atlas from Compose.

Makes a lot of sense in terms of pricing. Especially if you’re using WiredTiger.

Not having Oplog on the free tier isn’t the biggest deal.

Being down is obviously pretty bad. Your database was offline for the weekend?

Elie

20

Hey @a4xrbj1,

I’d like to point a few things out. First, the “free” MongoDB atlas is specifically listed as a “sandbox”. NOT a production ready, or even development ready instance of MongoDB. It is a free sandbox with the purpose of giving you a playground to test out Atlas, nothing more. And why would you expect 24/7 support for a free product? That seems like a really high expectation.

Secondly, I’ve been using Atlas since the day it was launched and has 100% uptime since. That on the lowest size that only costs about 55 a month.

Thirdly, your link to the jira ticket isn’t working for me so I can’t check it out.

But lastly, the thing that concerns me most is that getting data into Atlas is STUPID EASY. You can just connect any client and import. Studio 3T is my preferred choice. But even doing it on the command line is about as easy as it gets. If you are struggling with these very simple tasks you aren’t really in a position to advice other people on their use of a product.

Wouldn’t it be better to admit you need help and ask the community instead of bashing the product?

Have a great day!
Joshua