Caching records for performance and object equality

core
1

I have an expensive transform on one of my collections, so I want to cache the transformed records and only update the fields that have changed.

This also gives the nice property that Records.findOne(id) === Records.findOne(id) Which is nice generally and is also useful for React’s pure render mixin which looks for changes using simple equality.

Here’s a simplified version of how I’m doing this currently:

const _recordCache = {}

class Record {
  static load(data) {
    let record
    if (data._id) {
      record = _recordCache[data._id]
      Object.assign(record, data)

      return record
    } else {
      record = new Record(data)
      if (record._id) _recordCache[record._id] = record

      return record
    }
  }

  constructor(data) {
    Object.assign(this, data)
  }
}

const Records = new Mongo.Collection("records", {
  transform: Record.load
})

Records.find().observeChanges({
  removed: ({_id})=> delete _recordCache[_id]
})

This is working well but I need to know when an Item should be removed because a subscription was cancelled and it’s gone from the mini-mongo cache.

Is there a way to be alerted when that happens?

Alternatively, would caching transformed records be a sensible thing for meteor to do itself?

2

You can define an onStop handler when you make the subscription, so you will be able to respond to this event: http://docs.meteor.com/api/pubsub.html#Meteor-subscribe

3

@robfallows That’s when I would have to respond. I don’t see any way of knowing what records I can remove from my cache though. I need to find the ones that were loaded by the outgoing subscription and that aren’t covered by any other subscription.

4

Look for observeChanges

5

Maybe percolate:find-from-publication would help?

6

@lucfranken I’m using observeChanges in my example like so:

Records.find().observeChanges({
  removed: ({_id})=> delete _recordCache[_id]
})

That only covers cases where a record is actually deleted though. I’m wanting to know when a record has been unloaded after its subscription has been cancelled.

7

@robfallows Thanks for that link, I don’t think I would use it directly, but I think I could copy its style of tracking things manually with this.added and this.removed.

1 Like
8

Alternatively, would caching transformed records be a sensible thing for meteor to do itself?

Yes it does, in fact we use it a lot. When it’s not needed to run code again and the caching strategy is relatively simple it’s a good way to improve performance of an app.

I’m using observeChanges in my example like so:

Sorry missed that, was not visible without scrolling.

This is working well but I need to know when an Item should be removed because a subscription was cancelled and it’s gone from the mini-mongo cache.

Are you really sure this doesn’t already work? If the unsubscribe happens your find() will start returning nothing so it will start calling removed() in observeChanges.

That only covers cases where a record is actually deleted though.

It is removed from the local collection on the client. So that should just work out of the bug. If it doesn’t it seems like a bug. The only thing observe should do is checking which changes happen to the query handle is has.

It should work the same as the initial calls you get when you start running observe. Then you get all documents. On removing it should call removed for every document. No matter from which subscription they come?

9

You’re right! I thought removed() wasn’t being called but in fact it was a bug in the code I posted.

I go confused by the different signatures of remove() for .observe and .observeChanges.

The correct versions are:

Records.find().observe({
  removed: ({_id})=> delete _recordCache[_id]
})

or

Records.find().observeChanges({
  removed: (_id)=> delete _recordCache[_id]
})

Thank you.

10

Don’t you mean Records.find().observe instead of Records.find().changes ?

Great to hear your succeeding now!

11

Err yes I did. Edited.

1 Like