Need help I'm trying to hack my way throught meteor's subscriptions

help
1

Ok guys it seems I may have hit a few walls and I need your help on this, maybe we can brainstorm some ideas togheter.

This is related to my project called “grapher”. https://github.com/cult-of-coders/grapher

Context
When we currently “statically” fetch data like:

users: {
  posts: {
    comments: {
    }
  }
}

We do 3 db queries.

  1. We first fetch all the users (1 query)
  2. Then we fetch all the linked posts for all the users and reassemble them (1 query)
  3. Then we fetch all the linked comments for all the posts found and reassemble them. (1 query)

(don’t mind how this was done, if curious you can check the source-code, but the thing is it’s done and works smooth as butter)

So instead of 1 + count(users) + count(users)*count(posts) queries, we do just 3. This works very well and it has been proven to give 40x speed improvements for relatively small “graphs”, it seems that it’s way faster to do much less queries to the db, and run the assembly rather than blast the database.

Currently, queries can be reactive, but in the backend we use publish-composite package to do this. I think it can be improved dramatically for this, by applying the same principle.

SO! Here is where I want to go with this:
I want to apply the same principle, but to subscriptions. After that I want to create this: https://github.com/cult-of-coders/grapher/issues/52 which means that I would have the first graph fetched by methods, then I start to watch only what I want from it, like a conditional reactivity, and I can blend-in reactivity for the fields I want only, and also only if I want it, for a specific node, and then do some badass things like deep graph merging.

Just imagine. You have a social network, so you get all your posts, with all your likes, with all the comments, all comment’s likes, with all the comments files, and you specify at which level you want reactivity, and which fields. BOOM! This can be leveraged very easy by the server, if it’s smartly published

I am not yet sure for the “hypernova subscription” if it is going to be faster, I just assume. However I have to check to be sure, and see how it will behave in terms of performance with small, big, or deeply nested queries.

That is going to be the first step of the puzzle, once I get that going, I can use it as foundation for conditional reactivity graph.

Here is where my research lead me to:
WAY 1 Maybe instead of 1 subscription for the graph above. I create 3 and do something like:

usersCursor = Users.find(queryConditions)
postsCursor = Posts.find(aggreatedConditionsFilters)
commentsCursor = Comments.find(aggregatedCommentsFilters)

// a subscription waterfall
usersCursor.observeChanges({
        added,removed,changed: () => updatePostsSubscription()
})

postsCursor.observeChanges({
        added,removed,changed: () => updateCommentsSubscription()
})

Whenever anything is updated, we fetch the data graph locally, and make it somehow as a reactive data source, don’t know yet how, but surely doable.

WAY 2 Create a single subscription that does what I want to do in the backend, like:

// inside publish function
usersCursor = Users.find(queryConditions)
postsCursor = Posts.find(aggreatedConditionsFilters)
commentsCursor = Comments.find(aggregatedCommentsFilters)

play with observeCursors and use this.added, this.changed here.

Now here’s where I got stuck:
What is the most efficient way to do this ? This is only performance related only. I am basically rewriting Meteor’s publish methods, instead of returning a cursor, I write it from scratch. I want to know details on this, if someone can help me. Point me to some bits of code in Meteor. That would be very very helpful.

Guys, trust me on this, if we get the conditional reactivity graph up and running, this will open a new world for Meteor.