RedisOplog has received today an update that improves performance even further. Taking it to the next step into its evolution. It may sound weird but by offering to opt-out of race-condition protection, we can open the gateway to incredible speeds in sending updates to large scale clusters, without generating spikes in the database.
The greatest impact will be seen in very large clusters, every live-change implied a database request on the receiving end. Therefore, database gets bombed. Another problem that this solves, is allowing to send events where the listeners read from a slave database (which may have not yet have the full data)
Race conditions happen in extreme cases (almost never to be honest), and if it happens in most cases it’s not a big deal.
There isn’t anything out there that makes reactivity so simple and so scale-able. It’s crazy how powerful is the Meteor code-foundation, without it being written so well, redis-oplog may have never existed, or we couldn’t have full BC with it.
Hey @diaconutheodor, thanks for all your hard work on these optimizations.
Could you explain a bit more what “when it happens it’s not a big deal” means? What are the possible downsides of turning this on. I believe this would be used for a very large production-scale application, where “extreme cases” are everyday problems
Imagine you see number of likes on a Post, if you see 230 instead of 231. It may not be a big deal. You can always do some invalidation at a certain interval to ensure that your data is still up-to-date.
It’s not a big deal in most cases. I have never encountered a case (in the apps I’ve built,architected) so far where a race condition like this could have business impact. It can be in others for sure!
So 1: Even in huge apps, it almost never happens, it’s not like 1000 people are editing the same doc at the same time. If this happens maybe it’s time to rethink your db model?
So 2: If it happens in most cases it’s not a big deal.
You always have the option to bombard the db not a problem.
Socialize packages have Redis Oplog functionality baked in and activate if you have it setup. So this include feed, likes, comments,messaging and other.
I will have to look, what we need to update there (or can add to functionality) with the new release.
Used this as it made sense. Saw the race condition happen twice or thrice in the console message (I think when running cypress tests). Hopefully we reach the point when this is already a big deal (we are still far from it)
Well, not every Meteor application in production is just about some like where a wrong number isn’t causing much trouble.
Our app analyzes DNA sequences and makes decisions on the basis of the underlying data. If a race condition occurs and certain data is either stale or not even there, it unfortunately has big implications as it leads to wrong conclusions.
So I think for Meteor apps in production, it very much depends how important the consequences of some stale data would be.
Forgot to mention that we see these race conditions when we run our 1300+ backend tests - two of our test suites show failures regularly when we run them with all other tests but when we run them with .only they deliver correct results. So yes, race conditions happen.
