A crazy idea for Meteor React hooks - and Suspense

I have a crazy idea for an improvement for useTracker. If you followed the implementation of the hook, you may have noticed it wasn’t exactly straight forward to get all the parts working together, because of the way that React manages hooks. We ended up relying on a number of difficult to choreograph inline invocations and timeouts to make sure we aren’t running in to trouble with concurrency mode, adjacent suspense, or error boundaries.

First, what’s the problem? The problem is that Meteor’s Tracker sets up a computation - this pattern sets up observers to “reactive” data sources for you, so you don’t have to wire that up and manage it all, and remember to remove event listeners later, etc… SUPER convenient. But that computation is a “side effect” - which we cannot set up in a stateless functional component in the render body of the function, without getting deopted, and losing the benefit of concurrency mode (and risking memory and resource leaks). It has to run later in useEffect or similar to make sure it’s not getting instantiated 4 times, and never cleaned up. (We work around this by running it in render, but then carefully cleaning up multiple possible instances with timers, and resuming the one that gets mounted if we can - it’s complicated). This is true for every kind of tracker operation, not just computations which contain a subscription, because they all set up observers.

Our hook works well! But a pattern is emerging to be able to leverage React’s powerful feature - Suspense - for async data fetching that would make the implementation of our hook much cleaner, and allow us to better leverage concurrency mode. However there is a cost. Here’s what that pattern - Render-as-You-Fetch (using Suspense) - looks like in pseudo JSX:

<Suspense fallback={<h1>Loading profile...</h1>}>
   <ProfileDetails />
</Suspense>

(ProfileDetails would have either useTracker or withTracker integrated.)

So adding Suspense isn’t so bad - we might want that anyway, in cases where there’s a subscription! It’s a little weird though in cases where we don’t have a subscription. And it wouldn’t display for very long, probably under a second in most cases in concurrency mode.

But there’s actually another complication. In the React doc linked above, they suggest triggering “loading” in an event handler. I see two problems with that - one with the suggestion in general, and one specific to Meteor. In general, starting the loading in a click handler (or similar) is fine, except in cases where you might want to trigger load from a route change (which for me is almost all the time). In that case, you’d want something in your component hierarchy to do the loading. For Meteor it’s more of a problem than that though, because we don’t normally consider the computation separate from our specific useTracker instance.

(We could consider a “resource” API as suggested in the React doc, which decouples the creation of the computation from the “read” operation from the computation. I’m going to think on this more, but we may still want something like I’m about to describe anyway, for managing cleanup - which the React article does not describe.)

So, one solution to this, is we could hoist the creation of the computation up to a Context Provider. The Provider would be responsible for managing the computation, the useTracker hook would check status of that provider though the context API, and if it’s not ready, would throw to the Suspense handler. That brings our minimum pseudo JSX to:

<TrackerProvider>
  <Suspense fallback={<h1>Loading profile...</h1>}>
    <ProfileDetails />
  </Suspense>
</TrackerProvider>

And of course, we could add an Error Boundary in there too. Anyway, there are a few questions in here. Which component should contain the actual tracker function? Should that be defined in the ProfileDetails component (that’s how it’d be now), or should it just be defined in the TrackerProvider, and a hook used in ProfileData to “read” the state? It could still be in ProfileData, and we just pass that up to TrackerProvider - and that adds some other questions, like can 1 provider manage multiple Tracker children? If we might have more than one child which needs a Provider, how do we keep track of which is which (might require an id spec or something in the hook)?

Additionally, maybe we can make this even easier, and combine the TrackerProvider and Suspense in to a custom combo component. Again, it’s a little bit weird, intuitively, that we even need a Provider, for cases when we don’t actually have a long delay or a subscription. There are things with short initialization timings that are tricky - like, you probably want a delayed “fallback” display on your Suspense component from displaying, to avoid ugly flashes of Loading. A Meteor version of Suspense could probably help manage that.

I didn’t mention it above, but this is more efficient than the pretty good workarounds we have now in a very specific way. If you have an expensive Mongo query in your tracker function, that’s currently being run in-line with your render function, which can slow things down. Concurrency Mode will help disguise that, but it will run multiple times (that’s what concurrency mode does)! With a Suspense model like I’ve described above, the query would only run once when the component render is “committed” (when it’s mounted). It’s very efficient.

Really, the question is simply, is this too much API to require?

Ultimately, I’d like to create a way to make this work which leaves the current model in place when no Provider or Suspense is used, but use the more efficient Provider/Suspense model where its available.

Another option would be to try to implement this for the more specialized hooks I mentioned elsewhere - but there’s really no reason this can’t be done for useTracker.

15 Likes

In trying to work through an implementation of this, I came to a familiar problem. We need essentially a unique ID for any given instance of useTracker. The lack of a way to associate a single useTracker instance with multiple runs (in concurrency mode) is why we had to jump through so many hoops with timeouts to get a single instance while cleaning up garbage.

But I just thought of a simple way to get a unique ID - and I wonder if it would work. Basically, we can create a string of the reactive function itself with function.toString() and then append the values of deps array as JSON. That should be unique enough to not get reused instances. If that’s true, I can make useTracker much more efficient (and the implementation simpler), without even requiring all this extra API for a Provider.

What do you think, would function.toString() + JSON.stringify(deps) be unique enough?

To be honest I think providing a unique ID for every useTracker instance (by hand) is not a big deal. There are times when an anonymous function would be enough but you give the function a name just for code clarity and/or debugging purposes.
useTracker('getUsersQuery', () => { ... } ) seems good to me.

3 Likes

I would do this as a fallback if an ID is not provided.

Yes, I believe it should stay that way and also only one Provider would be better, as we already have one provider for other stuff.

I had a similar situation with an early version of a preprocessor I had written for Svelte to integrate with tracker. The early version would inject a unique id into the component for each reactive statement. Maybe a babel plugin could be written to generate the ids for useTracker when the user didn’t provide one? It probably would be challenging to do this with React since it is more flexible in how components are written.

I tried to just use a randomly generated id (using Random.id()) before remembering that there are various state data that could change and invalidate the computation. It’s a multi dimensional problem:

  • The user might change data themselves (update “deps”).
  • React will call the same code multiple times in concurrency mode, and we need a unique ID that maps between all the multiple invocations, not a unique ID for each.
  • (Those concurrent renders don’t have a method for cleanup build in - they are simply tossed to the ether - which is why we really and truly cannot have side effects. We currently use timeouts and careful choreography to clean that up.)
  • There may be multiple trackers coming in at once to a single provider from different unique component instances.

Setting an id manually is even a bit tricky, because you might be using the computation as part of a component in16 places on the page (and getting rendered 4 times each in concurrent mode). The user would have to add unique data like “deps” to that ID manually, which could be tricky to get right. Since they are already adding deps to useTracker I figured we can just use those (and make deps required for using the Provider/Suspense). We could do a hybrid where the user could name their useTracker instance, and then we append the deps JSON to that. I’d probably prefer overloading the second argument to achieve that:

useTracker(computationHandler, { deps: [... stuff], id: 'my-tracker-id' })

I have a couple other things I’d like to add to an options object like that, rather than adding additional arguments to the function signature. That could work though - an ID would certainly be cleaner than function.toString().

If you can get the current callstack, then the caller of the useTracker should be unique, given a file path + line number. I use this technique for a library in ruby, since ruby gives you this information.

Does JS have something similar?

For reference, this is the ruby code:

      # we want the erb file that renders the component. `caller` gives the file name,
      # and line number, which should be unique. We hash it to make it a nice number
      key = caller.select { |p| p.include? ".html.erb" }[1]&.hash.to_s

edit: oops, didn’t mean to post this as a reply to csiszi

In JS you can only get this through a stack trace IIRC

key = new Error().stack
1 Like

That stack would make it even more unique than func.toString + deps as JSON. I wonder if it’s necessary though? What’s the worse that can happen if the func.toString + deps is the same as another somewhere - it should produce the exact same output. Maybe cleanup gets tricky though.

I was thinking about this a bit more - if I keep a global instance manager, I have to make an assumption about some React behaviors, that I don’t know for sure. I have to assume that when a render is run in concurrent mode, however many times that at least 1 will be committed, so that I can rely on the cleanup of useEffect in that one instance. But I don’t know that for sure - are there cases in react where it might through away all the renders, and never commit?

Maybe it’s best to reserve this method only for cases where there is a Provider and Suspense. Then I can be sure of the lifecycle. I’m going to start with an implementation along these lines.

1 Like

Been a minute, but I was just thinking about this again. Here’s an example from React’s docs on how to do this:

// This is not a Promise. It's a special object from our Suspense integration.
const resource = fetchProfileData();
function ProfilePage() {
  return (
    <Suspense fallback={<h1>Loading profile...</h1>}>
      <ProfileDetails />
      <Suspense fallback={<h1>Loading posts...</h1>}>
        <ProfileTimeline />
      </Suspense>
    </Suspense>
  );
}

function ProfileDetails() {
  // Try to read user info, although it might not have loaded yet
  const user = resource.user.read();  return <h1>{user.name}</h1>;
}

function ProfileTimeline() {
  // Try to read posts, although they might not have loaded yet
  const posts = resource.posts.read();  return (
    <ul>
      {posts.map(post => (
        <li key={post.id}>{post.text}</li>
      ))}
    </ul>
  );
}

That resource idea is very similar in some ways to a computation, except with maybe an tiny extra wrapper on it to provide a get handle. Some of the example code could fit nicely inside a computation. One of the cool things the above example code does, is it Suspend at two different points based on two different loading conditions from one resource (or computation). It could probably do the same for errors as well, even though that’s not set up in the example.

One of the things the react folks suggest is kicking off loading as soon as possible, even perhaps in a click handler. For much of the Meteor code examples though, things are kicked off in the react lifecycle. So we’d need to think about how to build a resource manager to enable that. Something Redux like could work, but we can probably go a little bit simpler to start, by simply creating a Tracker provider, and having the computation method defined there, instead of inline in the useTracker hook. In the react component, we’d instead simply use the context value from the provider. This has so many potential benefits, while also vastly simplifying the implementation of our React integration. The downside is it requires a lot more set up by the user - they have to be aware of Provider, Suspense and ErrorBoundary - or we’d combine them in one unit which can serve as all 3.

I thought of trying to preserve the current pattern, where the useTracker computation is defined directly inline with the component which will use the data, by essentially passing off the reactiveFunc to a Provider/Suspense handler higher up the React tree. This could get tricky though, if a user tries to create multiple computations with one Provider.

I’m working on a sketch implementation of this and will try to get it up soon.

1 Like

This is turning out to be pretty crazy. The simple straight lines just don’t work. We can’t use the original idea of just storing the computation in a provider because we’d have to track multiple instances in concurrent mode anyway - the idea that we can create one, and just use that one when the single instance mounts won’t work because the computation encloses a set of state hook setters, and those may not be the right ones for the mounted instance. (There may be some hocus pokery we could employ to tap in to the single instance for changes using onInvalidate, then rebuild the computation on whatever the final mounted instance is in useEffect - maybe.)

We can hoist the definition location to the Provider, but that gets tricky with how to specify deps. I suppose we could just make it an attribute on the provider JSX, but that seems weird. Some other things about how to use Suspense and Error bounds produce interesting questions - like, how does React actually handle unresolved/unrejected promises in concurrent mode? Does a provider get rendered multiple times in concurrent mode, or is it deopted? Does the Provider always mount before children (so useEffect will have run) or can the Provider and children where useTracker might be used be rendered in that initial pass without either being mounted? I have no idea what the answers to these questions are.

What should probably happen is we completely separate the creation of computations from the use of them, by creating a Redux like reducer system, to define the “resources” and keep them out of the React tree. In Redux all reducers are combined in to a “root reducer” and consumed through the use of a selector hook (and “actions” trigger loads, and other state changes). That pattern could work here. (It’s sort of similar to my connector pattern, just with some early registration.) We could probably even make the provider easier to configure than Redux (and more code splitting friendly) by managing the registration of defined computations outside of the react tree - in our own non-react state.

The only question I have is, what would the consumption side of this look like? Specifically around how local state like document ids, etc. are passed to the queries for subscriptions and find operations.

One of the interesting problems comparing Meteor’s hook with the resource concept in the React examples is that the example doesn’t produce a live data feed - it just returns data at the end one time. In Meteor we have to actively manage a live data source, and update it if/when data changes (deps) - which makes everything tricky.

Meteor’s collection system actually already has some answers to these questions. A query is just a query - and I’m circling back to the idea that a general useTracker hook is what is making this all so challenging to implement. Separate useSubscription and useFind type hooks probably make sense here, where useSubscription (or a method call which loads data in to a client only collection or ground db) is the action and useFind (or useCursor) is the selector.

A useFind type hook, which wouldn’t be responsible for setting up subscriptions and all that, would be quite a bit easier to implement, even with Suspense and Error Bounds support, and it doesn’t even need a provider, or the redux type system (or, you could say it already exists in Meteor Collections and Pub/Sub). I’m thinking this is the way to go. It should be easy to set up suspense for data with useSubscription and maybe even in a generic way (any thrown Promise triggers Suspense) with useFind.

(This is all a little stream of thought - been on vacation, and just poking at this every so often. I think I’m getting somewhere though!)