Source announcement here:
Wondering how this will work with Meteor Mongo Driver and the oplog-tailing.
3 Likes
I’d guess this will be the problem.
1 Like
Yeah, I think so too since internally it is a different implementation, good thing we’ve redis-oplog.
1 Like
Was just about to post this… So do you think we could plug directly into this? I think it would make for a really lightweight server if I had my ec2 instance just plugged into their server.
Could you elaborate on why do you think this would make for a lightweight server?
Im imaging kind of like Dynamo DB it would be on a separate server as opposed to running my mongo DB on the instance where I’m hosting the web app.
I see, but you can do that with any MongoDB provider right now (Compose, Atlas etc…). Actually, I don’t think anyone should be running the DB server and the App server on a single machine for any production app, maybe for demos…
1 Like
I guess that’s just shame on me then… I’ve been doing it without an issue… But I also have little to no traffic still. I was concerned about security issues of having to open my DB to the internet… Is that not a concern?
It’ll for sure crush with little more traffic and you won’t be able to horizontally scale.
As for the security, well it’s a “secret” url that opens https connection with authentication, it’s as secure as your frond-end app. You can run a MongoDB cluster behind firewall if you want the extra security. But I really don’t see how a single machine can run any serious traffic…maybe you can cluster some app instances on a machine connecting to a local DB so that would allow you leverage all the machine resources up to a point, but then you’ll have to scale horizontally, so the DB has be moved out. Theoretically, I think you can hit around 2k+ concurrent users (with optimized pub/sub) using 32 CPUs if you utilize all the cores of the machine. However if you’ve traffic spike with heavy load and you’re using several user specific pub/subs on lading page, the 32 apps running on the those core won’t be able to handle that.
But back to the topic, MongoDB historically did have scaling issues although it improved over the years and I don’t know about the recent benchmarks. I think having an alternative implementation with the same API that claims to be scalable is good for market since it creates competition, it’ also good for Meteor since it provides us with more hosting options.
Looking through the supported APIs, it appears DocumentDB has no support for Mongo’s oplog, or change streams, which I guess is no surprise because change streams were introduced in Mongo 4, whereas DocumentDB copied the 3.6 API. So DocumentDB seems much less useful as a reactive data store than MongoDB.
In other words, DocumentDB is only a drop-in replacement for MongoDB if you weren’t using any of the features Amazon decided not to support. And in Meteor’s case the omissions are pretty important.
6 Likes
Might be able to use DocumentDB if you use redis oplog
2 Likes
There could probably be a good use case between Meteor remote collections and DocumentDB
Saw this announcement…
I like Mongo and haven’t ever had an issue with it. But always looking at playing around with and exploring new tech as well and this one is no different. I don’t think I have the chops to try and experiment trying to get this up and running without a working example though 
Since all but the latest Meteor versions expect Mongo <= 3.6, I assume there’s not much reliance on change streams? Although I know the oplog tailing is quite important. Do you mind describing the consequences if Meteor lacks access to oplog tailing? I assume Meteor still works, but it polls Mongo for updates?
The scaling issues should be mostly limited to DDP and reactive applications though. Swap out DDP and replace with JsonRoutes and triggers and polling, and things should scale pretty much the same as any other node app.
My guess is that DocumentDB is a move to support FHIR at scale, as Amazon/JPMorgan/Berkshire are going after the healthcare industry. They need a massive document store to go up against MUMPS and Intersystems/Cache.
5 Likes
This text from the DocumentDB FAQ suggests that DocumentDB might have an oplog API:
“Additionally, you can use most existing tools to migrate data from a MongoDB database to AWS DocumentDB, including mongodump/mongorestore, mongoexport/mongoimport, and third-party tools that support Change Data Capture via the oplog.”
That sound somewhat encouraging to me. If anyone wants to collaborate on some experimentation, let me know.
In a recent app I’ve built (almost done) I mostly don’t use pub/sub at all. The only real pub/sub I have in there is for user presence and auth, everything else is data delivered over methods, to avoid the overhead. How hard it would be to move just those collections over to something like this, and not worry about the oplog. I wonder if it could take enough pressure off my Mongo cluster (in Atlast) to reduce some scaling headaches/costs (if I have any, I know very little about those particular problems, since I haven’t had them, yet).
2 Likes