Meteor Scaling - Redis Oplog [Status: Prod ready]


#446

@dirkgently fixed it.

Released 1.2.2_1. And also wrote a test for it here: https://github.com/cult-of-coders/redis-oplog/blob/master/testing/publish-counts/client.js


#447

Wow thanks so much! I am going to talk to my boss about paid support.


#448

Released 1.2.3

  • Ability to customize functionality of redis failure
  • When redis connection resumes, all publications get up to speed with their latest data
  • Other bug fixes and improvements

#449

Guys I’m terribly excited about this: https://github.com/cult-of-coders/redis-oplog/issues/179

Translation of intent:
You can do whatever the heck you want with reactivity. And you can easily allow communication from outside services with your Meteor, as they can dispatch events to Redis.

This, I believe will complete the scaling of reactivity for good by allowing devs to taylor the reactivity the way they want it. You cannot get more performant than this, even if you start from scratch with express, websockets, redis, you will have roughly the same performance benefits. But this will save you a lot of time, as it’s so nicely integrated with Meteor and it’s secure by nature.

So we have 3 levels of scaling:

  • Add RedisOplog -> Already a huge benefit for high-traffic apps, especially for subscriptions by _id
  • Fine-Tune reactivity by using custom channels/namespaces
  • Do whatever the heck you want with Vent. (As it is not tied to Mongo, it will work with any type of database)

Again guys, you may not have read this :smiley:
it completes the scaling of reactivity for good

That was it. After this implementation, RedisOplog 's focus will be on perfecting itself, better test-suites, better code quality. And it will be LTS until 2024.


#450

brb, building the next Facebook to test this claim


#451

No more GraphQL or Apollo for me! Meteor alone is sufficient for my needs. Meteor now scales beyond our expectations. Thanks to @diaconutheodor and other contributors.


#452

Meteor = potatoes
Apollo = tomatoes
Meteor + RedisOplog = sweet potatoes

All we ever wanted was good potatoes, but then MDG gave us tomatoes. Now we can finally make some healthy french fries.


#453

Great work! I think with this flexible API and RedisOplog, you’ve closed the loop on all Meteor scaling issues. Furthermore, we can add external reactive data sources easily. I think this approach will offer the best of what the industry has to offer done the Meteor way.

I personally think this project needs it’s own marking page, and needs to be to recommended in the Meteor Guide so the community can organize around it. It also needs to be sponsored by large clients.

Once again thank you so much @diaconutheodor for taking this initiative, this is great!


#454

Dude, you simply rock!!


#455

Great news! I will definitely try this with my production app which as a high velocity of reactivity. Will probably try to do some tests in late Nov, early Dec. That will be the earliest chance I get.


#456

Great work! It seems that work on Redis Oplog is going full steam ahead. Considering this, might I make a suggestion, @diaconutheodor . The selling point of Redis Oplog is performance, but until each developer integrates this into their bottlenecked project and actually runs their own tests, most potential users of the package will have a hard time assessing the actual performance gains that are up for grabs. And as you know, the first click, or meteor add, is the hardest. To induce interest from a wider audience, perhaps a few general ‘before Redis Oplog vs after Redis Oplog’ benchmark statistics would be beneficial here?

Of course, each project is different with its own performance issues and synthetic benchmark results will probably be rather superficial, but some ready-made sample data is better than no sample data at all. And if the numbers do show a massive improvement in performance over the default setup, then showcasing said numbers, in bold, would seem to make sense.

For example, I found the view layer shoot-out by @mitar very interesting (https://github.com/mitar/meteor-web-rendering-framework-benchmark). Although it generated little feedback on the forums, I’m sure many found it useful, including me (by the way, thanks mitar for that!). Something similar for the data layer would be interesting.

Oh, and by the way, let it be mentioned that it is entirely possible that such data is already displayed somewhere publicly. Maybe even somewhere in this massive thread. I have not looked very hard as the data layer scalability is not currently the first priority for my own work.


#457

This new addition sounds wonderful and I’m trying to wrap my head around it. One of the things I’m exploring is using ElasticSearch as primary data store, instead of Mongodb. We need to do a lot of searching and complex queries and Mongo is just proving to be unpredictable. While we are not experts in ES, it does look more promising.

So I’m wondering if this will allow us to have reactivity (maybe not live queries) if we hook a publish into each entry point that modifies data. And the clients will listen as usual.

Any thoughts?


#458

@diaconutheodor What’s Vent? Can it be used currently with external data source with reactivity? A little note on Vent would be helpful.


#459

I’ll let @diaconutheodor keep me honest here, but if I understood it correctly, it’s a generic reactive Meteor API implemented using Redis at the app level (as oppose to DB) thus giving developers full control over their reactivity sources and avoid the log tailing performance bottlenecks. Furthermore, it’s in-line with how the rest of the ecosystem implements reactivity with all the leg work done the Meteor way.

With that you can start prototyping and even go-live with Meteor’s pub/sub and then when you scale and hit performance issues, enable Redis Oplog and if you still want more control then you can switch to Vent at the bottlenecks!


#460

Hello guys, some quick updates, I managed to completely implement Vent, alongside tests, and documentation. It’s not released yet because I need to clear out all the existing bugs and write some extensive tests.

Check-out the docs: https://github.com/cult-of-coders/redis-oplog/blob/feature/redis-vent/docs/vent.md

@msavin build it :smiley:

Just look at Grapher. It solves for Meteor & MongoDB what GraphQL solves for all the databases. Plus it’s reactive by nature.

MDG didn’t think of Meteor users when building Apollo, they thought of themselves, and honestly, Apollo is a beautiful concept, but the biggest issue they have is the “satisfy every database”. This is why with far less resources and Meteor, I managed to implement Grapher. Did I also Mention that Grapher can be integrated with other databases/api’s by using resolver links ? :smiley:

RedisOplog will soon reach maturity. The guys from classcraft are working on integrating it. They will be the best for showing what this baby can do. After that I’ll be continuing on writing a nice article on Meteor’s official blog, and stress them to put it in the official guide! :smiley:

I’m waiting on the guys from classcraft for this. They have a real application out there, so there won’t be “synthetic” benchmarks. They will be the key to show-case the benefits and I will help them with integration at every step.

To clarify, you will be able to make anything reactive. It’s just you have to write more code, to dispatch the events to redis and handle them.

Like air in a vent flows in a room, that’s how Redis Vent will push events if you subscribe to certain endpoints. You can already read the docs mentioned above.

Notes:

The main reason of redis-oplog, is not the fact it’s only needed by 5% of Meteor apps, it’s a selling point for Meteor. Imagine this scenario:

CTO: Hey we have this it’s very easy to develop on it, but if we reach 10k online users we may get into serious scaling issues, however there are other frameworks that are more scalable.
CEO: Fuck it, I’d rather have my back covered and invest more dev hours in something else.

We’ve lost many people along the way because of that. And the fight is not over yet, NextJS is an amazing tool in what it does, server-side react rendered headless pages load in an instant, even if it’s heavier for the server, there’s no connection kept active, there’s no big bundle to be downloaded to render your app. And most of the times, if you have a blog, or a shop where you display products, you don’t need reactivity/websockets there. NextJS is definitely a step back in terms of the evolution of the web (we’re back to server-side rendered apps), even if it solves elegantly some problems Meteor currently has, the future proof solution is to bring best of both worlds together in a beautifully unified framework.

I see the following:

  • Expose Meteor’s RPC (Methods) as a HTTP API
  • Conditionally start websocket connection when you need it only
  • Create a nice wrapper for server-rendered apps
  • Dramatically decrease the size of the necessary bundle for Meteor to run

There’s lots of work to be done.


#461

simple:rest does this in a zero-config approach, it’s great. I use simple:json-routes often for a basic CRUD endpoint


#462

Would love to see more MDG support here. I feel like they’ve never been involved in this project despite it’s massive importance to the community. Perhaps it’s because the money is pushing them towards GraphQL and Apollo.


#463

This is it! Exactly what I have been saying about MDG.


#464

@elie I agree that not only Meteor’s support, but any kind of support is beneficial here. I could not have done it without the support of many members here who helped me identify bugs, suggested improvements, etc.

It’s not wise to blame MDG because they were not involved in this, let them be focused on evolving Meteor’s foundation. I want them focused on that, here is our territory, we got this.

I’m like 1-inch close to finalise RedisOplog, the guys from classcraft are going to use it in production soon (1-2 weeks), they will be the best case study for all the Meteor community.

The future looks exciting.


#465

That’s an admirable attitude. Behind every great initiative, product or company there is someone with a great mindset :slight_smile: