@dinos and @jononomo and others that have found fault with forum post length, this should please. Below is my “thin forum post summary” for my “fat medium article” linked to at the bottom:
THIN SUMMARY:
The problem with Tracker is one thing (and by proxy, if we are being honest, the true problem with Blaze!):
tracker re-runs functions many times more than they need to be re-run (which is the sole cause of Blaze rendering jankiness!).
and a fair fight would be “Tracker 2.0” vs. Redux, and what Tracker 2.0 would have fixed is:
re-runs only occur if the precise dependent data utilized changes, not just because dependency.changed() was called
“property-level cursors” that automatically populate the fields option, eg: post.title within Spacebars automatically makes your fields option become: Posts.find(selector, {fields: {title: 1}})
an options param to Tracker.autorun(func, options) to further constrain what triggers re-runs. Helper methods could achieve something similar by wrapping your methods with a call that assigns options before assigning them.
once re-runs don’t run more than they have to, snapshotting Minimongo and ReactiveDict become a real option, and we can time travel (even if Minimongo is made of mutable state)
we need a Chrome devtools extension to visualize tracker dependencies between functions (graph diagram) as well as a 2nd visualization that depicts the stack frame of dependent autoruns sequentially. Picture “branches” of the graph highlighting as they update similar to the React devtools for components. Now you would know exactly what your autoruns are doing! We shoulda had this long long ago!! It should be similar to this execution visualizer for RxJS: http://jaredforsyth.com/rxvision
The in depth exposition, i.e. THE FAT MEDIUM ARTICLE is here:
If there seemed to be support in Meteor community to come together and create a managed pool of resources for things such as this, I would volunteer for the board and contribute.
Heck, I’d start a Meteor non-profit if others are interested for this purpose. Have a board of directors to manage the funds and allocate for projects such as the above.
Are you going to focus on building these for free?
Tracker 2.0
Blaze 2.0
Meteor specific Code-Splitting
Just a few questions:
Lets say you are. Are you going to consult with anyone on your ideas, get feedback from your peers, MDG before starting? What if by consulting with MDG before hand you’d stand a better chance of them incorporating what you build into core? What about working with others, have you asked others in the community for help with any of the above, if not, why? What about long term maintenance?
Also, are there any cases of Non-Profits maintaining open source software? It’s not just about paying for the tools, it’s managing the direction of the tools, it’s formal communication, it’s cooperation.
That’s my precise goal in discussing it so much in the forum as of late. I’m about to release #2 on your list there. It’s called Meteor.Component. It’s not very big–so sometimes the best way to converse is by throwing an actual concrete example out there, i.e. something you can immediately use rather than just theoreticals you can just talk about.
I think MDG has done themselves a big disservice by keeping all their “packages” under the main meteor/meteor roof. What simply needs to happen–well, in implementation its not so simple at this point, but its easy to describe–is: everything needs to be unbundled by default, so anyone from the entire NPM community can work on it any time they want. Currently nobody outside of MDG does much for their core packages. There’s only 16 developers with more than 50 commits:
community developer and Blaze Components author we all know, @mitar comes in at 48 commits. Right near him is founder @debergalis with 113. That’s compared to @glasser at the top spot with 3,386 commits and @dgreensp with @dgreensp in second with 1,880 commits.
In addition to being “unbundled” more, they need to be promoted. Their tools/packages need a life of their own outside of Meteor.
I’m suggesting the first step is their own repos, rather than live in the main meteor repo. It’s pretty obvious stuff. I wonder if it’s just “technical debt” that they never got around to undoing. There’s likely some technical reason having to do with automated builds and whatnot, but I doubt it’s an unsurmountable challenge. My opinion is that Meteor hasn’t been concerned with MARKETING anywhere near as much as they should be. In the developer world, marketing isn’t about big billboards on the side of the highway–it’s about nuanced things like repos that are easily accessible, i.e. easy to contribute to. Meteor is anything but easy to contribute to. Cuz its tools are so coupled and bundled!
They’ve taken great strides at the technical aspects of unbundling their packages, hence all the packages we have pre-added in our packages file which never used to be there. We just need to complete that unbundling.
The most important thing toward that aim is being able to live on “bare metal” NPM. To achieve that goal, we have to unbundle everything to begin with. In my Meteor 1.3 wishlist I discuss generating a main.js file which actually has the code which loads and instantiates all the stuff Meteor does for us behind the scenes on startup. This seemingly small thing is a big deal in terms of showing developers how all the pieces of Meteor come together. Combine that with easy access to the individual repos, and you now have a recipe for more contribution.
…So to answer your question, basically what I’m saying is that if you converse in public (which MDG doesn’t really do, which I do), and if you develop in unbundled way within githbub repos (like MDG doesn’t do and most us developers do), then we already have the “democratic” checks and balances you’re asking for. After all anyone can contribute to a github repository. And if your pull request isn’t accepted, but it was something the community wanted, everyone’s gonna hear about it in the ISSUES or associated forum. The most important keyword I think of when it comes to the future is “decentralization.” That’s what bitcoin is about. That’s what NPM is about. That’s where everything is headed. When you take on big chunks like Meteor has, decentralization is harmed. But when you take on just Tracker 2.0, well, you’re basically voting by doing. Others can vote “with their feet” by using it, or make bigger “votes” by discussing or contributing code.
In short, the “democracy” question is barely a worry when it comes to one guy. We just have this issue because MDG is so non-decentralized. But you’re right, I need to make sure I consult with them if any of this stands a chance of gaining adoption. As far as the rest of the community, that’s exactly why I’m writing all this. A lot of people seem to like it, and a few people have a hard time taking off their fresh functional blinders. The reality is I’m less worried about them since they already are getting their hands dirty in preparation to move forward. If those same people–who are generally outspoken early adopters and experts–can see the merit in things like Tracker 2.0 it will go a long way in terms of executing the practical strategies I’m proposing to improve meteor.
I really like the idea of making tracker and minimongo “reduxable”. Mutable state is really something we got shot by a few times.
So that would not only help react but all v-layer libraries. Though I believe it could be hard to fully erase the magic happening at the edge of tracker<->minimongo
And on ur medium article. I think basically Mobservable should replace Tracker. @sashko MDG should really take a look at this and let us know the pitfalls they see. There clearly will be some walls we run up against since we have a huge number of tools built on top of tracker. For its behavior to change in any significant way may cause things to break. But I think if we could prove Mobservable and the fixes it provides is a major enhancement we could get buy-in to upgrade tools that depend on it. Eg session, collection cursors, blaze helpers, etc.
I PM’ed faceyspacey some upgrades on tracker that I have been using at my company and he said that I should post them here. Here is the original message:
I read your Medium post about Tracker 2.0 and I thought that it was very insightful. I’ve got a couple helper functions monkey patched onto Tracker that I think you might find interesting. Here is a Gist: https://gist.github.com/veered/30ae6f79ec48506af80f
I think these helpers are a step in the right direction. And I think the code at the bottom (which I haven’t spent any time on improving after I had the original idea), could be used to build out way better debugging tools for Tracker.
When you are evaluating tasks.areEmpty you can stop at the State.get('tasks.items') and execute its modify first, make sure it is up-to-date and then continue with tasks.areEmpty. Functions are executed synchronously and only once. Perfect!
The problem with that is, once again: Meteor reactivity comes from many places, not only a single tree. For example, add another reactive source to the mix: Meteor.userId().
If you want to make this work synchronously, you need to know if Meteor.userId() has already changed or is going to change before executing any of the State modifiers.
The sad truth is: you cannot know. Tracker reactivity has no priorities and is spread in many places along Meteor. So the solution is what Meteor people already did: make Tracker async and wait for every reactive source to invalidate its computations. Then start executing computations.
For that reason, I don’t think Mobservable is useful or has any advantage over Tracker in Meteor if you don’t want to throw a lot of stuff which is already working with Tracker.
For Tracker, this means sometimes functions are going to be executed twice or even more times until they reach an stable state. That’s the reason MeteorFlux actions have three phases:
Register callbacks (sync non-reactive).
State.modifies or Tracker.autoruns (async reactive, it keeps executing functions until everything is stable).
AfterAction callbacks (sync after all reactivity has finished and is stable, non-reactive).
Explained like this, this looks like really complex stuff, but in the end coding with TRP is simpler and less prone to bugs. Most of the time you don’t have to know about how it works internally and that’s what makes Tracker and Meteor so simple to start with.
I really like your implementation. It’s similar to what I tried to achieve with my ReactiveState package. And the problem I wrote in the last post could be solved by your transaction pattern. Actually what Tracker does with its async approach is to make the current call stack a transaction and finishes it when everything has been executed and the CPU is idle again.
I have a question: why do think remaining sync is so important? Debugging, testing?
The other problem I had with my ReactiveState implementation was that I could only control what was inside ReactiveState, so reactive stuff outside the system broke the execution chain. But if instead of using a package on top of Tracker we re-write it (Tracker 2.0?) or find a way to use Mobservable instead, we could control the whole dependency tree and optimise the number of computations.
I think Tracker 2.0 could use two important performance improvements from Mobservable:
Make a proper execution chain derived from dependencies so computations are executed in a optimal order, not randomly like now.
Check for value changes in that chain so it can stop unnecessary executions.
I tried the first one and I am already doing the second one with ReactiveState:
// This retriggers when ANYTHING inside the post object change, not only title.
Tracker.autorun(() => {
let postTitle = Posts.findOne().title;
});
// But using ReactiveState to store the state...
State.modify('post', (state = {}) => {
return Posts.findOne();
});
// ...it retriggers only when the post title has changed.
Tracker.autorun(() => {
let postTitle = State.get('post.title');
});
Food for thought, really. Thanks for sharing all this amazing stuff!
Maybe community drive Tracker 2.0 can be implemented on top of mobservable.
Backwards compatibility relay in the fact that rerun behavior of actual implementation is a little “undetermined”.
Mobservable project go back us to good times of reactive data. I don´t really like the imperative aproach of ‘Flux actions’ and explicit ‘tree of truth’.
@luisherranz Could you elaborate a bit on what you mean by internal vs. external state? Mobservable doesn’t rely on stores, single state tree or other dedicated places to stores state like in flux architectures. Anything that is decorated using observable can be used a source of reactive data, no matter where the data originates from. So I don’t think your state / userid example above wouldn’t be an issue for Mobservable.
Sync behavior is quite fundamental to Mobservable (it is quite similar to Javelin in this regard); asynchronicity shouldn’t be used as means for scheduling; the only reliable source for correct execution order is the dependency graph. Once you have that, running stuff asynchronous only complicates stuff unecessarily; one can observe glitches, or introduce really nasty bugs like:
// pseudo code:
x = 1;
y <= x * 2 // computed value
x = 3;
if (y > 4) {
// will this code run? impossible to answer in an asynchronous world...
}
Note that running asynchronous doesn’t prevent glitches from happening. You still have to make sure that everything in the right order. So it doesn’t solve the fundamental problem of running the computations in the right order and minimizing the amount of computations. The only thing it solves is processing multiple state changes at once (which can be achieved by transaction indeed in Mobservable). Note that transaction could be applied quite generically, you could by default wrap event handlers in transaction for example.
Running all derivations synchronous makes it a lot easier to comprehend what is happening, is easier to debug and is very optimizable.
What I tried to explain is that when somebody is using Mobservable, it controls all the reactive/observable sources, so all of them can be controlled, executed, scheduled, whatever.
In my case, I was trying the same approach with a Meteor package only (ReactiveState), where I store the state of the app. The approach worked well for all the reactive sources of the package itself (internal), but not with the other Meteor reactive sources (external) because I couldn’t control them.
So what I tried didn’t work not for Mobservable itself, but because I wasn’t modifying Tracker, just enhancing it.
The solution to improve Tracker needs to be Tracker 2.0 and not only a Meteor package and after reading your inspiring article about the Mobservable implementation I think that might be the way to go.
Ok, I see. I like the transaction pattern but I can see a problem with everything-is-a-package or everythign-is-a-module architectures, where code runs from very different places. So you can’t just do a single transaction() function with the changes, they are spread over different modules. Default wrappers can be a solution but you have to implement them. That’s the good point of async execution, you don’t need to implement anything, it works in any case, although yes, it’s harder to debug, optimise and so on.
Note that it is harmless to not use transaction. Yes, it is a bit inefficient, but as long as your derivations are idempotent as they should be, you wouldn’t be able to tell the difference. So transactions can always be applied as an afterthought optimization, while the sync / async decision itself is quite fundamental. I think easiness / predicatability should prevail here over premature optimization
Edit: btw, note there is actually an autorunAsync, but that is a mere exception
