My experience with MongoDB Atlas - high data transfer usage

@csiszi
It really depends on the scale of your project and the type of queries. If, for instance, you have a lot of reporting, data drilling, big data, global reach etc you don’t want to run all those queries on your primary DB.
If your project is small, let’s say up to 5000 users you can just hook up Atlas in the same region (literally meters away) with your Galaxy deployment. For large projects you would be looking to data center architecture solutions like AWS EBS in front of Node servers and EBS in front of your Atlas VPC (you can read here about it: https://docs.atlas.mongodb.com/security-private-endpoint/ and you can also check on the network peering).
It is less important who is your Mongo provider as long as your MongoDB can sit in the same “LAN” with your Meteor deployment and have the features you need (like Oplog). Sometimes people complain about one provider or another without considering that their Meteor Node is running in Australia while the MongoDB was “transparently” deployed in Ireland with some … feasible and fairly-priced provider.
If you use BI or IoT, Atlas would also be your best option with their Atlas Stitch.
I think the old Meteor 1.0 question “Where should I DB” is today something more like “What exactly do I want to achieve”.
Large tech companies have a full department to deal with the architectures (infrastructure and DB). With Meteor and small teams we have to think that through ourselves.

Now … to answer your question … it is safe, it is recommended and it is advisable to anticipate your exact demand for computing now and in the future. If you do Galaxy in AWS, of course you will put your Atlas in AWS too, that is the only thing that really matters.

3 Likes

Thanks for the long answer. I expect maximum 3-4000 users so it’s definitely not large. My concern was that some folks reported a bug which caused extreme traffics for the mongo db even if the project was really tiny.
For another project I used compose, but they’ll end supporting “mongo classic” deployments in June.

1 Like

I am thinking that maybe that is not a bug but rather misconfigurations or geographical separation between Node and DB.
Until you hit 100 concurrent users you could stay on an Atlas M0 and upgrade as necessary. By that time you will know exactly how Atlas performs. In this case you will need to manage your own backup and later on migrations. For backups I keep an eye on this GitHub - stefanprodan/mgob: MongoDB dockerized backup agent. Runs schedule backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more. and planning to give it a try in the coming days.

I am attaching for you about a year of data from my Atlas on the project I work on. I only have test users, data is growing every months (this is a social network with posts and chats, contacts etc). But I want you to see that traffic volumes are very very small no bug, no leaks, I get exactly what I consume.

Something also to keep in mind is queries. Some people pull a lot of data (by mistake) map through it to get thousands of ids to use them to query the DB again - instead of aggregations and projections. All complex queries need to be aggregated on DB and only the results returned ( and convert into traffic).
Other people use some FileFS of GridFS etc like in-Mongo File storage and store images, video and large files. So, if you have a hero image on your home page and everyone gets it from your MongoDB, imagine the traffic.
Again :))… I was long but I am trying to say that it was probably not a bug … Just trying to raise your confidence … I don’t work for Atlas :slight_smile:

Screen Shot 2020-01-30 at 17.37.48

6 Likes

Atlas + Galaxy works but you have to be a bit careful with traffic because of the extreme expensive traffic pricing. Extreme accidental traffic bit me once. Check every .observe() call to be sure you don’t accidentally observe something big. Or don’t observe at all - instead, do whatever your observe() would do next to write calls instead. With observe, every write will be mirrored as traffic to observing app containers. Be sure to have monitoring and alerts for daily traffic switched on. And double-check you’re deploying to the same region! :slight_smile:

2 Likes

I don’t see the high charges that others have posted here. Our MongoDb Atlas runs on US East where also our Meteor servers are.

We do have a lot of traffic and queries running and we’re quite happy with the quality and knowledge of the Atlas support people.

For transparency here’s the current bill until the 24th of May:

Disclaimer: We’re part of the MongoDb Startup Accelerator Program though they manage to f$ck that part up (we never got any credits or where on any official comms until I complained to them last week - they claim they had experienced a problem “with our automated notifications system” and thus they didn’t know we never received any notification). That’s just as a side story.

1 Like

Do you have some docs on this DO setup?