Meteor Scaling - Redis Oplog [Status: Prod ready]

announce
41

Not for 2 updates. But only once for one update.
Let’s say I have a collection messages. I have documents like:

{
 _id:'foo',
 thread:'bar', 
 message:'hello world'
}
{
 _id:'qux',
 thread:'bar', 
 message:'hello mars'
}

And you have 2 publications to which you are subscribed that publish:

Messages.find('foo');

and

Messages.find({thread:'bar'});

If I update the message property of document with _id foo, I only want to get that update once over DDP.
With the redis approach you’ll have 2 messages: one on the messages::foo channel and on the messages::* channel.
Without mergebox you’ll send both of these to the client.

Not saying there are no solutions for this. On the client side I could save the state and check if something has actually changed when receiving a DDP message. Edit: this won’t work, since multiple updates could arrive in any order. So if you have 2 updates:

Messages.update('foo',{$set:{message:'bar'}});

and

Messages.update('foo',{$set:{message:'qux'}});

This can now arrive at the client as: ‘bar’-‘qux’-‘bar’-‘qux’.

43

You’re the man @seba, this is what I needed, another brain on me on this to challenge the approach, to help me craft it until perfection.

Ok, now regarding sending messages twice to the client, that refer to the same “document”. Thing is, some subscription may have some fields, other subscription may have different fields, the cost of processing this difference may be too much. And I personally think it’s not that big of a deal… to send it twice, because generally, you wouldn’t have such collisions (I may be wrong on this)

Currently we have a “merge-box” at publication level only. And if we want to cache same-type of subscriptions: https://github.com/cult-of-coders/redis-oplog/issues/5 this can open a can of worms if we have a merge-box at connection level.

Regarding making 2 updates like first I set something, then I set something else. It’s a very valid point, we have 2 options:

  1. Make sure that redis gives us the messages in order they were subscribed, queue-like
  2. Before sending it to the client, fetch it from db.
44

I’m glad you don’t take it as critique. I’m really excited to see where this goes. I’m just afraid that if you, like me, need the properties of single in-order updates, it quickly becomes very complex or not that better performing than the current solution. Doesn’t need to be a show-stopper, since a lot of users might not need this.

Now I’m absolutely no expert in this matter either. I did recently create something simple for apollo subscriptions:


This basically reads messages from the oplog and pushes them to channels similar to the redis channels you propose (but just in-memory publish-subscribe). That way subscriptions can choose which channels to listen to instead of running the query on all messages.

1 Like
45

I’m just afraid that if you, like me, need the properties of single in-order updates, it quickly becomes very complex or not that better performing than the current solution.

We just need to assure that messages from redis come in order, and processes are done in a queue, again in order, and problem solved. I will check your repo when I get the chance maybe I can find some interesting ideas in there.

46

UPDATE

BAM!
I completed the first phase of this project, which is: blindly coding my way through the basic features and writing some naive tests.

You can now clone this package, in your own project. I explain here how to install it and how to run the tests.


(Install redis first on localhost, we don’t have ability to configure it yet)

Now it’s time for unit testing, which may lead to better isolation of components. And then integration testing, and then finally find a way to properly test this client-side. This is going to take a while :slight_smile:

The PoC is up and running guys, you can add it and test it in your app. Most likely it will have some bugs!

After this is done:

  1. Find a simple way to disable reactivity.
  2. Same-subscription caching. When you have a blog and 100 people, you wouldn’t want 100 publications and “message” listeners for redis
  3. Tackle concurrent updates issue addressed by @seba
  4. Publish-composite replacement ? Or make a bridge to Grapher ?
  5. Write super efficient and memory efficient code.
  6. Start running some metrics comparing it to oplog.

We are finally making Meteor scalable.

13 Likes
47

Will you try to make this production ready or
is it just a POC ?

48

@doedel

Production ready is on the way :slight_smile: that’s the plan! But it will take time to label it “Production Ready” now you can help us with ideas, try to break it, etc. Any help is good help. I am basically trying to do what MDG guys did with oplog, they had geniuses working on it…

This will have great impact on everything. After we’re done and stable with this, I think it’s best to abstract the MongoDB driver so you can have reactivity with ANY type of DATABASE (but it will depend on priorities, currently focusing on Meteor + MongoDB). But I believe this is going to hit hard in the JS community. I really hope it will convert a lot of people to Meteor.

4 Likes
49

Sounds good and your are right: Its up to the community to support guys like you.

My first contribution will be to try your package in
a test project and see how it works so fare :slight_smile:

1 Like
50

So, I created a simple starter project for redis-oplog:

2 test are ok and one failed in the client.

Before you can use this project you have to install redis:
brew install redis
and start the redis server.

I will now try to use your package…

51

Hey, they are failing because of fixtures. You need to restart it, I know, very poorly implemented. But I mentioned this already :smiley:

52

I’m glad someone is actually working to solve this problem finally - so many people were waiting on MDG to make the move, and MDG seemed reluctant as their focus is on Apollo at the moment, so I did not expect any of this work until Apollo at the earliest.

Thanks so much from the Meteor community for working to solve one of the few, biggest problems Meteor has had!

2 Likes
53

It was about time, right ? :slight_smile: Well we shouldn’t blame MDG for that, they gave us Meteor, one of the most beautiful things, it’s up to us, the community to return the favor and give them a helping hand!

Thanks for the kind words :slight_smile: we’ll make this redis-oplog perfect, and maybe offer an integration with apollo ? We’ll see!

Cheers

3 Likes
54

Just wanted to chime in that I’m following the thread intensely and appreciate your work in hopefully getting this out as a solution.

What would really help to involve others (me at least) is if you could draw out your solution in visual way (schema/flowchart). I’d be happy to contribute where necessary but an overview of the core idea of your solution would speed up understanding.

5 Likes
55

Very well, I understand, we definitely need a good schematic.
The problem was that until now, things were volatile in terms of how to design this.

I now have a clearer view and I can do some blueprints. And make a plan so the community can help (Slack channel maybe?)

Can’t offer a date, but will do. thanks for wanting to help, it is more than welcome.

56

I don’t exactly “blame” MDG - it’s just their priority order at work - which should be understandable for any developer, especially in their position.

It’s only that, with much of the changes being worked on now, I would have personally preferred upgrades to the scaling over most of the future updates. But I do understand the importance in their priority as well.

57

Thinking of your namespaces concept: your example is chat but a more enterprise one is multi-tenancy right? One namespace per company for example? Or is the amount of namespaces limited?

1 Like
58

Cool work there! Quick question; how do you plan on getting non-meteor mutations into redis? What will happen when db is, say, manually updated?

59

You read my mind :slight_smile: i actually thought about this today as an advantage but first I thought about chat… so yes it is very possible to use for multitenamt systems

60

Cool work there! Quick question; how do you plan on getting non-meteor mutations into redis? What will happen when db is, say, manually updated?

No plan yet. Since this will be very customly tailored on the dev needs, I will first need to think of some wild usecases, figure out what they have in common, create a facade that help you with these common tasks.

If db gets manually updates, nothing will happen unless you publish the message to redis. The price we pay for gaining control of reactivity at app level.

1 Like
61

Redis doesn’t have an oplog native to the database like Mongo does. So like @diaconutheodor said, there are always going to be tradeoffs. This shouldn’t be a problem for most applications.

I’d rather have the ability to scale my app without non-meteor reactive mutations (only useful for maintenance), than to not be able to scale my app with non-meteor reactive mutations.

Scaling is the pain point and the problem here. I’d say that takes precedence over edge cases.