@dirkgently fixed it.
Released 1.2.2_1. And also wrote a test for it here: https://github.com/cult-of-coders/redis-oplog/blob/master/testing/publish-counts/client.js
7 Likes
Wow thanks so much! I am going to talk to my boss about paid support.
8 Likes
Released 1.2.3
- Ability to customize functionality of redis failure
- When redis connection resumes, all publications get up to speed with their latest data
- Other bug fixes and improvements
2 Likes
Guys Iām terribly excited about this: https://github.com/cult-of-coders/redis-oplog/issues/179
Translation of intent:
You can do whatever the heck you want with reactivity. And you can easily allow communication from outside services with your Meteor, as they can dispatch events to Redis.
This, I believe will complete the scaling of reactivity for good by allowing devs to taylor the reactivity the way they want it. You cannot get more performant than this, even if you start from scratch with express, websockets, redis, you will have roughly the same performance benefits. But this will save you a lot of time, as itās so nicely integrated with Meteor and itās secure by nature.
So we have 3 levels of scaling:
- Add RedisOplog -> Already a huge benefit for high-traffic apps, especially for subscriptions by _id
- Fine-Tune reactivity by using custom channels/namespaces
- Do whatever the heck you want with Vent. (As it is not tied to Mongo, it will work with any type of database)
Again guys, you may not have read this 
it completes the scaling of reactivity for good
That was it. After this implementation, RedisOplog 's focus will be on perfecting itself, better test-suites, better code quality. And it will be LTS until 2024.
19 Likes
brb, building the next Facebook to test this claim
3 Likes
No more GraphQL or Apollo for me! Meteor alone is sufficient for my needs. Meteor now scales beyond our expectations. Thanks to @diaconutheodor and other contributors.
3 Likes
Meteor = potatoes
Apollo = tomatoes
Meteor + RedisOplog = sweet potatoes
All we ever wanted was good potatoes, but then MDG gave us tomatoes. Now we can finally make some healthy french fries.
6 Likes
Great work! I think with this flexible API and RedisOplog, youāve closed the loop on all Meteor scaling issues. Furthermore, we can add external reactive data sources easily. I think this approach will offer the best of what the industry has to offer done the Meteor way.
I personally think this project needs itās own marking page, and needs to be to recommended in the Meteor Guide so the community can organize around it. It also needs to be sponsored by large clients.
Once again thank you so much @diaconutheodor for taking this initiative, this is great!
3 Likes
Dude, you simply rock!!
1 Like
Great news! I will definitely try this with my production app which as a high velocity of reactivity. Will probably try to do some tests in late Nov, early Dec. That will be the earliest chance I get.
1 Like
Great work! It seems that work on Redis Oplog is going full steam ahead. Considering this, might I make a suggestion, @diaconutheodor . The selling point of Redis Oplog is performance, but until each developer integrates this into their bottlenecked project and actually runs their own tests, most potential users of the package will have a hard time assessing the actual performance gains that are up for grabs. And as you know, the first click, or meteor add, is the hardest. To induce interest from a wider audience, perhaps a few general ābefore Redis Oplog vs after Redis Oplogā benchmark statistics would be beneficial here?
Of course, each project is different with its own performance issues and synthetic benchmark results will probably be rather superficial, but some ready-made sample data is better than no sample data at all. And if the numbers do show a massive improvement in performance over the default setup, then showcasing said numbers, in bold, would seem to make sense.
For example, I found the view layer shoot-out by @mitar very interesting (https://github.com/mitar/meteor-web-rendering-framework-benchmark). Although it generated little feedback on the forums, Iām sure many found it useful, including me (by the way, thanks mitar for that!). Something similar for the data layer would be interesting.
Oh, and by the way, let it be mentioned that it is entirely possible that such data is already displayed somewhere publicly. Maybe even somewhere in this massive thread. I have not looked very hard as the data layer scalability is not currently the first priority for my own work.
1 Like
This new addition sounds wonderful and Iām trying to wrap my head around it. One of the things Iām exploring is using ElasticSearch as primary data store, instead of Mongodb. We need to do a lot of searching and complex queries and Mongo is just proving to be unpredictable. While we are not experts in ES, it does look more promising.
So Iām wondering if this will allow us to have reactivity (maybe not live queries) if we hook a publish into each entry point that modifies data. And the clients will listen as usual.
Any thoughts?
@diaconutheodor Whatās Vent? Can it be used currently with external data source with reactivity? A little note on Vent would be helpful.
Iāll let @diaconutheodor keep me honest here, but if I understood it correctly, itās a generic reactive Meteor API implemented using Redis at the app level (as oppose to DB) thus giving developers full control over their reactivity sources and avoid the log tailing performance bottlenecks. Furthermore, itās in-line with how the rest of the ecosystem implements reactivity with all the leg work done the Meteor way.
With that you can start prototyping and even go-live with Meteorās pub/sub and then when you scale and hit performance issues, enable Redis Oplog and if you still want more control then you can switch to Vent at the bottlenecks!
Hello guys, some quick updates, I managed to completely implement Vent, alongside tests, and documentation. Itās not released yet because I need to clear out all the existing bugs and write some extensive tests.
Check-out the docs: https://github.com/cult-of-coders/redis-oplog/blob/feature/redis-vent/docs/vent.md
@msavin build it 
Just look at Grapher. It solves for Meteor & MongoDB what GraphQL solves for all the databases. Plus itās reactive by nature.
MDG didnāt think of Meteor users when building Apollo, they thought of themselves, and honestly, Apollo is a beautiful concept, but the biggest issue they have is the āsatisfy every databaseā. This is why with far less resources and Meteor, I managed to implement Grapher. Did I also Mention that Grapher can be integrated with other databases/apiās by using resolver links ?
RedisOplog will soon reach maturity. The guys from classcraft are working on integrating it. They will be the best for showing what this baby can do. After that Iāll be continuing on writing a nice article on Meteorās official blog, and stress them to put it in the official guide!
Iām waiting on the guys from classcraft for this. They have a real application out there, so there wonāt be āsyntheticā benchmarks. They will be the key to show-case the benefits and I will help them with integration at every step.
To clarify, you will be able to make anything reactive. Itās just you have to write more code, to dispatch the events to redis and handle them.
Like air in a vent flows in a room, thatās how Redis Vent will push events if you subscribe to certain endpoints. You can already read the docs mentioned above.
Notes:
The main reason of redis-oplog, is not the fact itās only needed by 5% of Meteor apps, itās a selling point for Meteor. Imagine this scenario:
CTO: Hey we have this itās very easy to develop on it, but if we reach 10k online users we may get into serious scaling issues, however there are other frameworks that are more scalable.
CEO: Fuck it, Iād rather have my back covered and invest more dev hours in something else.
Weāve lost many people along the way because of that. And the fight is not over yet, NextJS is an amazing tool in what it does, server-side react rendered headless pages load in an instant, even if itās heavier for the server, thereās no connection kept active, thereās no big bundle to be downloaded to render your app. And most of the times, if you have a blog, or a shop where you display products, you donāt need reactivity/websockets there. NextJS is definitely a step back in terms of the evolution of the web (weāre back to server-side rendered apps), even if it solves elegantly some problems Meteor currently has, the future proof solution is to bring best of both worlds together in a beautifully unified framework.
I see the following:
- Expose Meteorās RPC (Methods) as a HTTP API
- Conditionally start websocket connection when you need it only
- Create a nice wrapper for server-rendered apps
- Dramatically decrease the size of the necessary bundle for Meteor to run
Thereās lots of work to be done.
9 Likes
simple:rest does this in a zero-config approach, itās great. I use simple:json-routes often for a basic CRUD endpoint
2 Likes
Would love to see more MDG support here. I feel like theyāve never been involved in this project despite itās massive importance to the community. Perhaps itās because the money is pushing them towards GraphQL and Apollo.
1 Like
This is it! Exactly what I have been saying about MDG.
@elie I agree that not only Meteorās support, but any kind of support is beneficial here. I could not have done it without the support of many members here who helped me identify bugs, suggested improvements, etc.
Itās not wise to blame MDG because they were not involved in this, let them be focused on evolving Meteorās foundation. I want them focused on that, here is our territory, we got this.
Iām like 1-inch close to finalise RedisOplog, the guys from classcraft are going to use it in production soon (1-2 weeks), they will be the best case study for all the Meteor community.
The future looks exciting.
9 Likes
Thatās an admirable attitude. Behind every great initiative, product or company there is someone with a great mindset 