Full-stack reactivity: opt-in or opt-out?

community
62

That was a good read vlasky, thanks for sharing.

1 Like
63

I guess developers should pick the stack that best suits their project? Different databases for different use cases.

The API could accept a reactivity flag:
“realtime” or -1 = Full reactivity, if the database has a reactive method/driver, e.g. oplog tailing. Otherwise, throw an error.
0 = none, load once.
Any other number = every x seconds.

And then a config can set a default to make queries terser and please the “reactivity by default” and competing crowds.

64

I think from an economic perspective, reactivity being off by default and opt-in makes more sense.

It shouldn’t be the case that all data is reactive by default because we just haven’t solved the problem of scaling 100% reactive applications yet.

Like Sashko said, the beauty of the Prius is that it’s still gasoline powered by default, with electric capabilities, in an infrastructure that isn’t yet completely electric.

Toyota didn’t jump and make the switch to a primarily electric vehicle right off the bat. Why? Because at the time, the technology and infrastructure weren’t ready for it.

They decided to take a look at where the market was, understand what people could afford to buy (because big batteries don’t come cheap
 and neither does scaling reactive applications), and they gave the older gasoline guys a chance to adapt to what will soon be a predominantly electric market.

Remember, gasoline is the heart of the Prius, not electricity (even though we all know it’s better).

Reactivity is no different. We’re at the dawn of a transitional period. This stuff is all very new, and not just to developers, but even to users.

Think about it
 should we have a primarily electric car, when we haven’t fully figured out the technology yet and batteries are still crazy expensive? Or should we instead have a primarily gasoline car that the current infrastructure can manage, that people are comfortable with, and that also boasts electricity when you need it?

I just don’t think it’s a hard decision to make. Believe me, I think we can all agree that reactivity is the future, just as much as we can all agree that renewable energy, electric appliances, and artificial intelligence are as well.

But it’s not hard to see that even in those fields, whenever we’re at the dawn of a transitional period we support both technologies. And the older technology is always given priority over the newer technology, until we finally reach the dusk or twilight of that transitional period.

Eventually, it’s out with the old and in with the new until the cycle inevitably repeats it self, next time with higher standards and more knowledge brought to the table.

TLDR: renewable energy is great, but the market doesn’t yet support it completely. We still give priority to older, more scalable systems of infrastructure. Reactivity is no different. It’s expensive, and the market/infrastructure isn’t completely ready for it. We should still give priority to older, more scalable systems of infrastructure, while developing these technologies in the background. Eventually, reactivity will overtake nonreactivity similar to how renewable energy will overtake nonrenewable energy. But it all starts with giving the older system priority (because that’s what economically makes sense and because we’re still at the dawn of a transitional period), while still making way for the new.

65

The format of the poll though. I bet many clicked opt-in but did not read the part that says by opt-ing in you agree to switch off reactivity by default.

1 Like
66

So Apollo isn’t “the” data retrieval solution for scaling with reactivity?

68

Right, it’s possible, but what do you do, store the retrieved data in a Session or ReactDict as the case may be, via the method callback?

69

My two cents: optimistic ui, reactive updates between the ui layer and the client cache should be on by default. Anything involving comms to the server should be opt-in.

But please do it via different classes or methods (with names something like “reactiveQuery”, 'staticQuery" etc rather than an option that has to be set with the method call. I’m guessing 80% of meteor developers (not the experienced ones) don’t even know about the option to do it with Mongo queries.

1 Like
70

Apollo allows you to choose which data is reactive, therefor allowing you to scale. The problem of scaling 100% reactive applications has not yet been solved, not even by Apollo. Even Firebase has a cap at 10,000 maximum concurrent users.

This thread is asking whether data should be reactive by default, and opt-out, or vice-versa.

71

Whatever data type is appropriate - those options you have mentioned or even just storing the method’s result value in an ordinary JS primitive datatype or object.

72

I think it is that simple. Live query for backends that support it, polling for those that don’t. People should know what they’re getting into either way. We’ve been running since 2012, had very few problems with scaling but we’re only scaling to max 30 users. Our app is business-to-business. Each client has their own deployment per business.

Not every app has a target market of 100k users. For ours having everything reactive by default makes sense. For others it doesn’t. So if you get to choose on day one, and not have to think about it every time you make an endpoint, I think that makes the most sense.

1 Like
73

Just, this won’t get us the smooth minimongo-api to fetch the data inside our helpers. My point, it feels hacky and unMeteor retrieving data this way.

74

We did a test with rethinkdb and meteor, one core for db and one core for meteor. 50,000 cursors updated in under 10s (has been a while - maybe even under 5s)

Disclaimer: indexed query

75

I’m not sure what you’re trying to say.

What do you mean by that?

76

You said 10k users seems like a bar, likely due to resources; so I am saying we can reach a lot more using rethinkdb as it is explicitly designed for reactive web / apps (however, rethinkdb still has a few kinks) – imagine 4 cores = 200k cursors, if each user has 2 - 4 subscriptions you can push up the bar dramatically.

77

Ah, I see. With regards to the 10,000 user limit, I was talking specifically about Google’s Firebase project.

As for Meteor, that’s not bad. I’m curious now - can you do the same thing with MongoDB (and have you done any testing similar to the testing that you have done with RethinkDB?

78

I have not done mongo testing. It would be interesting - so might do it.

I do know that theoretically there is a limit. When you reach a large enough number of cursors open in parallel, oplog trailing will stall, as each new instance of meteor will have to sift through all the logs to detect cursor changes. What is that number? Don’t know yet :slight_smile: but do expect it to be relatively low from the performance we are seeing. And graphQL won’t solve that, as it’s a db issue. The only real solution, I believe is polling, which is also very expensive, but does not have that theoretical limit. Bad or worse – take your pick :slight_smile:

1 Like
79

The Prius metaphor is interesting, but brings up another option. With the Prius, the operator does not decide if and when to use electric or gas, the framework (the car) does it automatically based on what is most efficient at the time. It would be interesting to have the framework make the decision based on the most efficient manner given the circumstances. And then you could still have a switch to manually choose one or the other. This model implies some AI sauce that makes the decision and possibly toggles it off and on based on network, server and client loads/circumstances.

3 Likes
80

So if somebody is a Prius buyer, he/she wants to take advantage of modern technologies (recuperation, electric motor) but don’t go full electric because he/she fears:

  • batteries are not dense enough
  • charging the battery takes longer than taking gas
  • charging plugs for electric cars are rare

Beeing a Tesla buyer means you don’t give a shit. 10’000 Tesla buyers make Tesla innovate further and overcome what we had problems with.

Questioning opt-in or out is like buying a plug-in-hybrid. Continuing to innovate and make reactivity more scalable is the only solution to make a step forward. Realtime everywhere olé :slight_smile: