My experience with MongoDB Atlas - high data transfer usage

#1

Just wanted to let everyone know my experience with MongoDB Atlas and see if anyone else has the same experience. I’ve been running my DB through MLab for the last couple years on their shared plan. This has been fine while my total DB size was around 1GB (3GB total counting the replica set), but with some new added features to the app the DB has grown to about 2GB (6GB total) which was causing me to have to compact the DB often to keep it under the 8GB limit. The next tier on MLab was $180/mo, over twice what I was paying, plus they still were not on mongo 3.4. I’d hears some good things about atlas, where I could get 40GB total space for $.1/hr for a 3 replica set, which was about equal to my MLab bill. Moving over was pretty easy using their built-in tools. Relaunched all my microservices with the new DB without a hitch.

The problems started almost immediately though. The first day I was hit with a $5 “AWS Data Transfer” fee. Atlas claimed that I used over 66GB of data on that day alone at $.09/GB. I never saw this kind of data usage with my app before. The app itself consists of 5 microservices and a handful of client connections. It does update data on a minute-by-minute schedule, but only a handful of items. The next day was worse. Atlas reported that I had used 99GB of data transfer for $9. The next 129GB and the next four days it averaged in between 99-120GB a day. The first week I had wracked up $50 in data use charges. At that rate my use would have been double or triple a dedicated plan on MLab was. I asked Atlas about it but they said it was my app was generating that data.

Out of desperation I created my own 3 member replica set on Digital Ocean using their $20/mo server (3X for $60 a month) which was a lot easier than I though to set up (in retrospect I should have done this sooner). Same level of security as my Atlas deployment. Moved my DB over. Atlas reported a max thruput on the database at over 10MB/s, with an average of around 3.5MB a second (as reported using MongoDB Compass). On my DO deployment the max as like 500KB/s, with an average of only 61KB/s. Many factors lower than Atlas was reporting and more inline with what MLab was reporting.

For now I’m going to stick with my own deployment. Much cheaper and backups don’t cost and arm and leg. It doesn’t have all the fancy reporting tools that Atlas has, but performance is the same or better.

Has anyone else seen these kind of data transfer fees on Atlas. I want to recommend it to other clients, as I like that it is on a current version of Mongo and their toolset is better than MLab, but can’t in good conscience do it if data usage fees will kill them.

6 Likes
#2

I have experienced the same, with a small DB being used for an app in staging that has only been tested very sparingly. One month recently my bill shot up 300% with these phantom transfers. I will be engaging with Atlas customer service soon and will try to follow up here.

2 Likes
#3

We’ve had Atlas on our possible upgrade list for a while now so always good to read about other people’s experiences. Thanks for sharing that @cspecter and @rtcarroll07, if they do get back to you with an explanation I’d love to hear it.

3 Likes
#4

I had Atlas contact me yesterday after I had cancelled my account. They wanted a post-mortem on what went wrong. It would have been nice of them to show that much interest when I was having the problem and still on their system instead of just telling me “the problem is with your app”, but better late than never. I couldn’t really do much but let them know the setup on my app and the kind of performance I was getting on my new setup. Comparing the readings using their MongoDB Compass tool (which is great and let’s you connect to any MongoDB install) of my new DO deployment vs. what Atlas was telling me is like night and day. My DO deployment is averaging around 64k/s including data between the replica set itself. Atlas was reporting 10MB/s, several times larger. I had migrated over from MLab where I had only used about 20GB of data transfer for the whole previous month whereas Atlas was reporting like 99GB/day.

Long story short, they don’t know what the problem is. They seemed to think that it was related to Meteor, but I don’t see how that is possible without the same problems showing up on other hosts. That or it was some interaction with Digital Ocean, which also seemed wrong. I feel like it is a mis-configuration on their part. If you plan on using Atlas I would be very cognizant of your data usage, and check it every day. But, to tell the truth, it was so easy to set up my own replica set on Digital Ocean that I can’t see using any of these MongoDB hosts for anything but the largest or most demanding deployments in future projects.

#5

@cspecter I have been using Atlas and Meteor since Nov 2016 and have not seen these types of issues. We have a heavy workflow of batch processing that lives outside our Meteor deployment with a database size of roughly 3GB. We moved over from compose due to similar issues you mentioned at the top of the thread. We are fully deployed in AWS and leverage Atlas’ VPC Peering for db connection. If it helps, our current deployment of Meteor is 1.4.1.1

#6

I am on release 1.5. Maybe it really does have to do with Digital Ocean interacting with AWS, as they suggested. I’ll have to check that out when I get a chance.

#7

We just tried migrating a Mongo database into MongoDB atlas and ran into all of these same issues.

Our database is literally 2 MB at the moment. Yet, we got charged for 1.139GB of data transfer two days ago and 2.425GB of data transfer yesterday despite barely anyone using the app.

When we opened a support ticket this is what we were told:

As is described in the Data Transfer section of the Billing documentation, we charge for each hour of Atlas server usage only when servers are active (and at a reduced rate when they are subsequently paused). A group with no deployment (zero servers running) will not incur any charges. The hourly cost per server varies based on instance size, disk speed, region in which the instance resides, and the cloud service provider the cluster is deployed to. There is also a baseline level of communication between the nodes in your deployment that is used to maintain high availability, which will also factor into your Data Transfer costs even if you have no active operations being performed.

Within MongoDB Atlas, we have various tools that are monitoring each cluster and are communicating with our Atlas application instances that are hosted in AWS in the us-east-1 region, which accounts for the ~13KBps rate of network activity that you can see in the YOUR_DATABASE Network metrics for yesterday when the cluster was active. Accumulated over the course of 24 hours, this adds up to about ~1.139GB of data transfer (which matches the usage shown in the project’s Usage Details).

As far as I can tell, we seem to be getting charged for both the servers in the replica set communicating with each other, plus the monitoring tools that make the Atlas dashboard possible. No wonder y’all have seen such extreme amounts of data usage.

3 Likes
#8

NodeChef MongoDB hosting does not charge for data transfer, offers advanced features and cost effective.

#9

Excuse me, I don’t know any experience with Mongo Atlas.
The price for create new one Cluster is for one database name.
Or we could create more database on One cluster???

#10

Yeah, that is about what I experienced. I’ve tried out all the major MongoDB hosts and all have their flaws/extra costs. As I said in my OP, I’ve found the best solution so far is to roll your own replica set on DO, at least depending on your project, DB size and client. Some clients are willing to eat the added expense, which is fine, but the project I was working on above had some pretty large data sets (financial data) that would cost several hundred (or more) a month on most of these services. It didn’t really take that much effort to get it running and you get so much storage and bandwidth for cheap. You don’t even have to forego a good management UI as Mongo’s Compass App works with all Mongo deployments. I liked Atlas’ features, but their customers must only be large companies with all that added expense vis-a-vis the data transfer fees.

#11

Have you tried out the ScaleGrid MongoDB hosting platform?

  1. No data transfer charges
  2. No backup costs
  3. Full admin/oplog access
  4. Digital ocean support
  5. Bring your own account on AWS/Azure

(Disclaimer - I am the founder of Scalegrid.io)

1 Like
#12

FWIW, my production app switched to Atlas in January. We have a specific use-case where we need to crank up to their higher performance servers at times (something we couldn’t do on Compose).

I do see the AWS Data Transfer (Same Region) and (Internet). The “Same Region” last month was 193.71GB @ $0.0100 / GB = $1.94 for the month (pennies a day). The “Internet” was 0.06GB @ $0.0900 / GB = $0.01 for the month. It’s always around this same cost each month. My app doesn’t have a ton of data transfer, so this seems reasonable.

My production databases is 429MB. Total disk usage of the cluster is 5.8GB. Running on the M30 size.