Meteor Scaling - Redis Oplog [Status: Prod ready]

announce
64

Updates:

I finished caching same subscription for many users on the same instance. Like when you have a blog page, or you have 5 people in a chat thread, if they are connected to the same instance, server-side there will be a single publication, and when the last observer leaves, it gets destroyed. All in a synchronized queue. I believed this was critical to performance. The overhead is really small, just a search over some strings, and done once when publishing.

I consolidated fine-tuning. Separating concerns for “namespace” and “channel”. More in the README.md file.
https://github.com/cult-of-coders/redis-oplog

I will drop “publishComposite” as a feature, at least for the first milestone, the reason is that it can lead to badly optimized code, and implementing an API to allow you to make it performant would have lead to a ton of work.

You can still have “first level” relations by returning an array of cursors:

Meteor.publishWithRedis('list', function (listId) {
     return [
           Lists.find({_id: listId}),
           Todos.find({listId})
     ]
})

Or if you use Grapher links module :wink:

Meteor.publishWithRedis('list', function (listId) {
     return [
           Lists.find({_id: listId}),
           Lists.getLink(listId, 'todos').find()
     ]
})

I kinda ignored testing (shame on me) So the focus now is:

  1. Unit testing
  2. Integration testing

Help me break this baby! I’m sure it has a lot of bugs and inconsistencies, we begin the last step and the hardest : making this rock-solid.

2 Likes
65

I’d also love to hear @sashko reply to this, as well as to my own alternative data architecture I proposed. I know they have their heads down coding making things Awesome, but while we’ve been waiting, community proposals have emerged that would make it great to see a Venn Diagram with Apollo that I’d love to see illustrated.

2 Likes
66

I think it would be nice if someone of MDG could tell us why they didn’t choose this approach a while ago. Are there any bigger disadvantages (except of reflecting third party updates)?

1 Like
67

I haven’t had some time to look into this in depth, but overall using Redis to handle subscriptions sounds like a great idea, especially if it allows the Meteor server to consume fewer messages. The goal of Apollo is to make reactivity entirely optional, but I suspect the way invalidations are sent around will be pretty similar with both.

2 Likes
68

@XTA
maybe it was just a bad design choice. People sometimes make bad decisions.

It seems @serkandurusoy was right, it was Arunoda who first thought about this, and he advocated that it helps scaling in the blog posts… It actually did to certain degree compared with poll&diff.

Now on the other hand, Apollo is not focusing on reactivity at all! So in the end, the blend of these two technologies will be a winner. However, Apollo is a data system, the mergin of the 2 would imply what exactly ? I’m confused.

@deanius
What did you propose, where ? Also, a Venn Diagram representing what ?

@sashko
It can consume less messages if you fine-tune it. However:
https://github.com/meteor/meteor/blob/master/packages/mongo/oplog_tailing.js#L38 oplog, like our redis subscribers, we listen to all incoming things from a collection. We can match them using “sift” library which is very fast. It’s the one I used in Grapher, previously was relying on minimongo, but it was very slow compared ti sift.

So if you want a publish for all users, or all users with some filters, the number of messages consumed by a meteor instance would be the same. However, the key difference here is the fine-tuning part, meaning you can fully disable reactivity, you can have it on separate channels only, making live chat apps possible.

3 Likes
69

So, trying to understand where the advancements are, to get this more clear:

  • Webshop with products: No real advantage
  • Forum topic list: No real advantage
  • Chat: Lots of advantage because more specific listening to only relevant changes
  • Apps, like New audio demo reel service built with Meteor: ReelCrafter , huge advantage because only listening to changes on the data for the specific user?

So that last one could be implemented as this:

Which means you could create a new namespace in every collection for every account. So instead of searching constantly through all tracks you now just load the namespace for your account. Correct?

70

I only know SIFT as a CV algorithm, but I take it you mean something else?

71

Hi @lucfranken

I said the number of messages consumed will be the same for those cases. However there are certain things that will give huge impact on performance in some scenarios. Will explain below.

Webshop with products: No real advantage

Partially true, because we added a “shared publication” principle which works at instance level. What it does is if you return a cursor or array of cursors, it takes the filters and selectors from those curors (and does a JSON.stringify) and makes a “key”.

What this means:
If you have 1 instance and 100 people are on the products list. On page 1, we have 1 single redis listener and 1 single processor which dispatches “added”/“changed”/“removed” to all observers (clients). If they are on page 2, like different limit/skip, they won’t share the same processor. The overhead for this is very small, just a search done once when subscribing, and when it stops, just a check to see there are no other observers so it can destroy. Sharing is done per publication name.

Also last night I woke up at 4 AM with another idea for large-scale, sharing redis “listeners” at instance level: Created an issue for this here

However, if you have a big shop, and you have let’s say 100 instances, chances of people viewing the same thing on the same instance are small. This is why in the future I will be seriously thinking of caching the state into redis: Created an issue for this here

These 2 will be critical. The problem is man, I don’t really care about large-scale that much, most projects won’t reach it, or they may not be designed for that, but even for a small project, instead of having 20 instances to handle 1000 online people, you would reduce it to 1 server with 8 cores 16gb ram, and use pm2 or something :slight_smile:

Forum topic list: No real advantage

Same advantage as above.

Chat: Lots of advantage because more specific listening to only relevant changes

True

Apps, like New audio demo reel service built with Meteor: ReelCrafter , huge advantage because only listening to changes on the data for the specific user?

True, but not only that kind :slight_smile:

Which means you could create a new namespace in every collection for every account. So instead of searching constantly through all tracks you now just load the namespace for your account.

Yes yes, the idea for that was to enable multi-tenancy, but ofcourse it can have other applicability as well. If you boot-up an invoice app, where you can have multiple teams, those teams live in isolation to each other, therefor, when performing mutations you specify their namespace, so listeners will be hitted only with relevant data. You could create an additional abstraction layer to do this automatically for you, so you won’t have to do it on every insert/update/remove :slight_smile: you could even override how the package works I will have to think of some ways to make it super-hackable so anyone could do anything they want with it.

As you can see moving away from oplog, to something customizable, opens the gateway to new technologies and ability to fine-tune the reactivity however you wish. If you trully want something large-scale ~50,000 online people, you would use only ‘channel’ publications, and implement custom publishers to redis to achieve that. Things will definitely be different.

4 Likes
72

I used it in grapher for data assembler:
https://www.npmjs.com/package/sift

At first I was using local collections to do that, but it was killing the cpu, worked with arrays and sift and gained I don’t know about 20-30x speed. It was ridiculous for big arrays.

73

I’m not sure how you mean ‘Reactivity optional’ - Using Tracker is one thing, but reactive programming, or ‘streams based’ programming (as opposed to imperative), is that not present in Apollo? I mean—the UI is subscribed to updates to the store so I think it is in reactive in a pure sense, but maybe not in the Tracker sense.

74


But I guess you have already read it.

75

That’s Apollo and GraphQL though. It’s not our traditional Meteor where everything is seamless and easy. Don’t get me wrong, I like the idea of Apollo - but it’s not as entry level, as easy to communicate, and as simple as Meteor is at Meteor’s current state.

This implementation would theoretically allow me to connect a Redis database to my Meteor app, and POOF scalability (more so than I would have had before).

2 Likes
76

We made it guys, we made it to a first release as a team!
This thread started 11d ago. We brainstormed together, we shared ideas, and now we’ve launched.

Read more on https://github.com/cult-of-coders/redis-oplog

It’s still not fully unit-tested. But it covered the essential functionality for a modest first release. Regarding benchmarks comparison, we need to do it in order to see which are the heaviest actions.

I need to stop iteration on features and perfect this. It can take 2-3 months until maturity ? I have no idea, we’ll see.

13 Likes
77

Great job, I will test it in a very busy application.
Do you have any benchmark static on achieved performance?

78

Not yet, did not have time for that, will post here when i get the chance. The real gem here is fine tuning reactivity and ability to perform changes without reactivity.

1 Like
79

Just couldn’t resist:

Had to add this one last feature, to make trully useful. Meanwhile, did anyone try it yet ? It’s just a plug-in plug-out in your app, I’ve tried it in my apps, it seems to work as expected, but I would like you to add it to your apps as well, and see if it breaks!

Thanks!

80

This is really interesting. Can’t wait until I have time to try it out.

1 Like
81

Going to try it out in the next few days. I’ve been following this thread closely and it’s exciting.

1 Like
82

I’ll be trying it in my app sometime within the next month or so when my schedule clears up. This is a really exciting alternative to oplog as well as Apollo, thanks!

1 Like
83

@jeremy_s

Going to try it out in the next few days. I’ve been following this thread closely and it’s exciting.

Glad you see it that way. I want bugs!

I’ll be trying it in my app sometime within the next month or so when my schedule clears up. This is a really exciting alternative to oplog as well as Apollo, thanks!

Not an alternative to Apollo, but yes it can make Meteor apps super-performant. Thanks!

1 Like