In our case everything ia working fine, too. Our instances stopped crashing when running the daily cronjob which deletes about 100k docs/day.
“We did not notice any major bugs” does not sound too assuring are there some kinds of small bugs?
Haha true, so, there were some bugs indeed after we went to PROD, but we solved them, and they were not MAJOR just some small edge-cases. Currently it’s stable.
Ok, so it sounds like there are no known bugs with the new version. Nice.
I have been using redis-oplog in production for months with hundreds of concurrent users. We had one small issue which was fixed immediately by @diaconutheodor. All in all I’m very happy and can recommend redis-oplog!
I’ve been thinking… few things that I think could be very synergistic with this package, and perhaps practical since we are connecting our apps to Redis:
- Creating/exposing Redis APIs for Meteor developers use, and to make it work over regular pub/sub. Redis is blazing fast, and can persist, so might as well embrace it all the way
- Creating some kind of load balancing solution around Redis, in the spirit of meteorhacks:cluster.
- Creating a “patching” API for redis-oplog in case MongoDB updates happen outside of the Meteor application. Maybe some kind of protected REST API.
Combined, they would take this package from a solution to scale oplog to a solution to scale Meteor, all built around the magic of Redis.
And one more: the idea of Redis going unavailable and stopping updates is a bit unpleasant. In the beginning, it would be nice if there was some kind of mechanism for developers to manage these scenarios or to force it to re-run all the queries.
I’m very likely going to start integrating this into my production app in a few weeks. One question came to mind (sorry if it’s been answered): you remove
oplog from your app logically, but what happens to Meteor’s backup “poll and diff” mechanisms? Do they still run with
redis-oplog? If the “poll and diff” caused performance problems before adding
oplog will it still when using
redis-oplog? Is there a way to disable “poll and diff”?
A new release will be coming soon making it even more stable. (1.2.1 I plan on finishing it this month)
Regarding your question, there is no fallback with redis-oplog. If redis dies, bye bye reactivity, if redis comes back alive again (we have a 30s infinite retry of connection) reactivity will be back up again. However, when redis resumes, we have to requery all observableCollections and apply changes:
However, you can still rely on poll-and-diff if you want: https://github.com/cult-of-coders/redis-oplog/blob/master/docs/finetuning.md#fallback-to-polling
You can even have mongodb oplog, redis-oplog and poll-and-diff at the same time by not overriding publish function, but I wouldn’t recommend that though.
redis-oplog is becoming a critical piece for the community, especially production apps. I recall there was planning (months ago!) for a blog about it on the main Meteor blog site (@diaconutheodor and I talked about it privately too and I offered to help review it).
Any updates on that?
Case in point: we now have thousands of concurrent users internationally, with servers in US, Europe and Middle East (planned), redis-oplog is planned to be the glue for that
Thanks for the reminder Ramez, planning o. Giving that draft feedback by
end of the day.
@diaconutheodor Would it be worth raising a PR on https://github.com/englue/meteor-publish-composite to see if @reywood can bring the adaptation you made for optimistic UI into the main publish-composite package? That would save you from maintaining a separate package and users having to change over to it.
There have been some recent updates to the package by @reywood, so we can benefit from those when using Redis Oplog without any extra maintenance headache on your part.
I tried and hated that cluster package. It had ingenious elements, but was very hacky. Load balancing should either be hardware-based or with the web server itself (e.g. nginx pro or tengine).
Unless I misunderstood what you meant.
@Longmate in the meantime I know found another solution without changing the publish-composite package. So I will deprecate my package and make it compatible.
Hi @thea, are we end of day yet?
Just joking, I am sure you already reviewed the draft. Would love to see that blog post.
Does anyone know if we can use Redis version 4 with this package ?
It’s available on compose.io
Why not redis cloud by redis labs… free plan should work great since there’s nothing actually being stored in redis.
I don’t see any changes inside the pub/sub so yep, it should work. Let me know if you encounter issues.
@copleykj that could work, but I suggest keeping your redis in the local network, for trully fast responses.