Using multiple databases with one Meteor application (RethinkDB, Elixir)

Hi,

The app I’ve been working on has seemed to have hit a MongoDB bottleneck. It’s a fantasy sports website providing real time drafts, but when 100 drafts are going on at the same time with around 1000 concurrent users, it seems like the database can no longer handle this load.

Ideally we’d like to be able to handle 1000s of concurrent drafts in the future, but I don’t think the current Mongo/Meteor setup could ever handle it.

Would it make sense to just scale part of our application to use another database such as RethinkDB. Does it become a mess to use two different databases for one website. What are people’s experience with this? And is it worth switching over the entire application to use RethinkDB? (I don’t even know what sort of scale Rethink can handle and what issues we might face with that).

This whole process should be made a lot easier with the switch to Apollo, but I do wonder about having to keep the databases in sync with each other (or if that’s even necessary).

Another idea I’ve had is to use Elixir and Phoenix framework for the drafting part of the site. Has anyone tried this? The idea would really be to use Elixir as the backend for drafting instead of Meteor. I don’t know if that would solve the Mongo problems although it may.

Anyway, interested to hear what people think.

Three words: Apollo, Apollo, Apollo.

I don’t think you read or understood… perhaps my fault for being unclear… but thank you anyway…

I didn’t try it yet, but in your case, it is 100% possible that Mongo causes the problem, you shouldn’t forget that Mongo isn’t a realtime database and Meteor’s LiveQuery is something like a hack to get realtime updates.

I would recommend you to move over to RethinkDB since it is a fully realtime database. The big problem here is that there is no featured Meteor package available, only one outdated that may work (you’ve to test it). There is also a blog post about Meteor and RethinkDB with an example.

Some possible solution would be to leave all user account data on a MongoDB database (because the core accounts package relies on Mongo) and save all other live data to the RethinkDB database. In the future, Apollo will solve this issue.

If you are thinking about moving to another framework like Phoenix, you may also check FeatherJS.

1 Like

I ran across this this yesterday, it’s called Feathers.js, very cool. Seems like it will help with scaling. Also, Apollo might help with scaling, either way you might need to do a rewrite.

We are in the process of working on a fork of Meteor for RethinkDB instead of Mongo. Preliminary looks good, we expect it to be ready in a few days.

Sounds great. We also have a chat application which we want move over to Rethink. I am just waiting for Apollo subscriptions and Meteor 1.5, but if we could get Rethink intrgration earlier that would be great. Does the fork work comfortable with Minimongo?

Yes, mongo syntax all the way. We like it more than ReQL!

Any update on this?
Does it uses slava’s implementation? And if Slava needs to wait for this: https://github.com/Slava/meteor-rethinkdb/issues/24 why don’t you?