Ok guys, so in order for this to get the desired adoption, we need publishComposite, for sure, it's not optional anymore and I understand this.
Because most of the apps that are in production use publishComposite (including ours), we cannot simply say "just rely on oplog for publish composite" or polling!
The problem of publish composite is, ofcourse, performance. It'll be a CPU & Memory hog:
For: users -> posts -> comments. We will have to instantiate 1 observable collection for users, lengthOf(users) observable collections for posts, and lengthOf(allPosts) for comments. If you have 10 users, each user has 10 posts and each post has 10 comments. We'll have: 1 + 10 + 10*10 = 111 "publications". Which is going to be roughly the same of having 111 subscriptions. Crazy.
The first idea I wanted to implement was doing an aggregation of filters at each child-node. Example, instead of creating lengthOf(users) independent ObservableCollections for posts, just do a single one, where filters represent all filters in an $or array. This idea had it's birth here: https://github.com/cult-of-coders/grapher/issues/36 . Do what we do for Grapher non-reactive fetch (assembly and deassembly) do for publishComposite.
The problem is that it can work with Grapher, because the way of getting links is known, the fetching strategy is immutable so we know how to choose whether we use aggregate framework or not. However, in this case, publishComposite would give users ultimate flexibility in terms of displaying any type or related data.
Aggregation of filters concept will fail if we have "limit", "skip" in child cursors. We cannot fetch'em all in a single query. Especially when these can change in certain children, because we have a function for each child element.
The options I see now:
- No limit/skip in child nodes (So we can aggregate the filters)
- Implement it bare-bones like I described above. (It can still be more performant than the current publishComposite)
- Move the composition client-side. (The problem with this is that it can lead to a lot of security wholes and a lot of boilerplate code, I know this because I solved this issue in Grapher using body exposure, it's crazy how deep things can get)
- What if we could stop reactivity at a certain node ? Example I want all users with all posts with all comments. I maybe don't want reactivity for comments. Or I want it only for users, or only for posts and comments. The higher the reactivity stops, the faster it is. But then the question arises, why use publish at all if you want to stop reactivity ? Doesn't seem like a good thing.
I will continue with option #2 now, just to have the ability to say farewell to MongoDB oplog.