Redis vs MongoDB OpLog Tail

SkyRooms · July 27, 2017, 4:36am

I’ve been looking for a comparison, can’t find much

I’m not even sure what Redis is, there’s so little information on it. Is it MongoDB + extras?

Is it FASTER than MongoDB in OpLog configuration?

Can I host my own Redis server? I currently use Studio3T for DB admin, what do I use for Redis?

Is it worth switching to?

raphaelarias · July 27, 2017, 6:07am

In general people do report that using it makes the meteor app more scalable.

Redis is a different DB. You can use RedisLab or other services like Compose.io. You can host it yourself too.

martineboh · July 27, 2017, 6:44am

The current Redis implementation of Oplog, redis-oplog lets developers control reactivity efficiently - this has given back reactivity to the app’s control. Meaning you can now build all kinds of apps from games, chat, and other collaborative apps without much scaling issues as opposed to using the mongo oplog for reactivity, which is the default behaviour of the Meteor reactivity stack.

Note: A cache layer can probably be better implemented in Redis. For more schema-able data, MongoDB is better. [Note: both MongoDB and Redis are technically schemaless].

SkyRooms · July 27, 2017, 4:14pm

Cool, I’ll check it out. Guess I won’t know until I switch over. Sigh… more work after putting 30 hours in to learning MongoDB hosting and Oplog trailing techniques…

hwillson · July 27, 2017, 7:18pm

I know this isn’t usable yet, but I’m just throwing something else into the mix. Mongo 3.6 will have a new Notification API - as in a real notification API that will support subscribing to collections (and handling events) out of the box. Depending on how this turns out, Meteor might be able to move away from tailing the oplog. This really could be an incredible step forward. It’s definitely early days still, but super exciting! A FR is in place here, if you’re interested in following along:

efrancis · July 27, 2017, 8:34pm

From the MongoDB Jira issue:

Finally, the oplog is a mongod-wide component which under heavy traffic means I may be tailing much more data than I actually need if my application is focused on just one collection or DB.

Ideally, creating an activity stream that is optionally published out would expose this through an API layer independent of the oplog. This would allow for applications like this to no longer directly rely on the oplog.

If we could also filter on the activity stream similar to how audit filters are applied it would allow for only reading the changes you care about as oppose to filtering on the application side and transmitting a ton of data needlessly.

Wow! I’d never heard of this before, but it sounds like exactly what Meteor needs for scaling! That gets me pretty excited

efrancis · July 27, 2017, 8:36pm

People have been using regular oplog tailing with Redis for years and it’s worked for them. Redis is just another option for (possibly) better scaling, at the cost of more manual and conscious choices around reactivity. No need to go down that path until you hit scaling problems in production which necessitate a change. Premature optimization is often a waste of valuable time that you could instead be spent building a product.

alawi · July 27, 2017, 8:37pm

First time I hear about this, wow, this is a major step forward, super excited to see this happen!

SkyRooms · July 29, 2017, 1:05am

Thanks @efrancis for the info.

However, I’m STILL not clear on what Redis is. Does 1 instance of Redis automatically oplog tail? Or like MongoDB, do I need to setup 3 instances to create the tail?

I’m currently experiencing database issues with performance, I built an MMO on Meteor. www.StarCommanderOnline.com - you’ll notice connection problems as the server for 10 consecutive people takes 14+ gigs of RAM.

I suppose I just to try it. Bleh.

vigorwebsolutions · July 29, 2017, 1:12am

Redis is a data structure store, often used as a database, but in the case of redis-oplog, used primarily for it’s messaging capabilities. Check out the link below for an overview:

github.com

cult-of-coders/redis-oplog/blob/master/docs/how_it_works.md


## How it works

### Sending changes

This package will allow you to use Redis pub/sub system to trigger changes. Any pub/sub system will work. We don't use Redis as a database,
we do not store anything in it.

We override the mutators from the Collection: `insert`, `update`, `upsert` and `remove` to publish the changes to redis immediately after they have been
sent to the database.

Let's take an example:
```
const Items = new Mongo.Collection('items')
Items.insert({text: 'Hello'})
```

After the insert is done into the database, we will publish to redis channel "items" (same name as the collection name) the fact
that we did an insert, and the *_id* of the document we inserted.

This file has been truncated. show original

SkyRooms · July 29, 2017, 3:15am

Thanks for the link.

So this looks EXACTLY like MongoDB in OptLog configuration.

So if I were to self host a RedisDB, would I also have to put the thing in a tri-cluster?

vigorwebsolutions · July 29, 2017, 3:26am

There are a number of threads about people’s experience with redis-oplog – it seems that anecdotally, a small DO droplet is usually sufficient to run the redis server.

XTA · July 29, 2017, 5:32am

Oh, you can also create a replica set with only 1 instance to enable the oplog. No need for 3 servers/instances.

If you want to go the Redis way, you only have to install redis-server on your machine. That’s it.

SkyRooms · July 29, 2017, 3:46pm

I think that’s what I was looking for.

It seems to me they perform at LEAST the same, and evidence for Redis is higher.

The fact I don’t need 20gb of RAM to run the DB server will pay for itself. I guess that’ll be my project coming up, indeed.