Efficiency for publish with regards to infinite scrolling / pagination

It seems like for infinite scrolling, you essentially publish a larger and larger cursor each time. is this efficient? does it not actually send the data again over the wire (unless it has changed?) Thanks

Hi @meowzus
You are right. It sends the full mongo fetch every time you resubscribe. Moreover it triggers to many actions for remove, fetch and add portions of data every subscribe. And it could be deadly slow to re-render after resubscribe.

It’s better to use methods (if reactivity isnt your aim) especially w React and virtual lists like react virtualized. You can simple fetch a chunks of data for pagination or infinity scroll.

// Client
// -------------------------------------------------------------

class MyList extends React.Component {
  constructor() {
    super();
    this.state = {list: [], count: 999999, hasMore: true};

    // we could use debounce to prevent accidental fetch
    this.fetchData = _.debounce(this.callRemote, 300);
  }

  componentDidMount() {
    // first fetch, let it be 50 records
    this.callRemote(0, 50);
  }

  callRemote(start, end) {
    if (!this.state.hasMore) {
      return;
    }

    Meteor.call('fetchDocs', start, end, (err, res) => {
      if (!err) {
        const list = _.union(this.state.list, res.list);
        const count = res.count;
        const hasMore = count > list.length;

        this.setState({list, count, hasMore});
      }
    });
  }

  itemRender(idx) {
    const item = this.state.list[idx];
    
    return (
      <div key={item._id}>
        {item.name}
      </div>
    );
  }

  render() {
    return (
      <VirtualList 
        data={this.state.list} 
        rowRender={(idx) => this.itemRender(idx)}
        rowsCount={this.state.count} 
        onScroll={(start, end) => this.fetchData(start, end)}
      />
    );
  }
}

// Server
// -------------------------------------------------------------

Meteor.methods({
  fetchDocs(start, end) {
    check(start, Number);
    check(end, Number);

    const skip = start + 1;
    const limit = end - start + 1;

    const cursor = DocsCollection.find({}, {skip, limit});

    return {list: cursor.fetch(), count: cursor.count()};
  },
});

What if i need some level of reactivity, such as if I have a list of items and i need them to be able to be liked, etc. Would this approach still work?

There are a lot of possibilities how to make some “light” reactivity. But you could reload chunks by timeout (as meteor handle updates on heartbeat) or after current user actions (like, reply, edit)

If you are interested in GraphQL, its common approach to fetch data back to client after mutations (user actions)

If you first subscribe with a limit of 50, and then change it to 55, the client will only retrieve the 5 missing docs. The server will still do the full 55 doc fetch though.

It’s not optimal to just increase the limit though, because eventually you’ll end up with a very big subscription. Instead you can keep the same limit, but add “skip” that you increase as the user scrolls.

I’ve created a demo of a pretty smooth infinite scrolling dynamic subscribe 50.000 document table here. Repo here

Oh, interesting. So it is true that at least sending over the wire is efficient right? That seems to be good enough for me, at least for now.

Say I wanted to abstract your example more, where for a new scrolling list all I would have to do is replace the “fetchDocs” call, while maintaining all the other logic, such as calculating the count, hasMore etc. Do you have a decent way of doing this? I can’t seem to come up with a decent idea other than using inheritance, which seems to be discouraged when using React.

Nice example.
But dont you think subscription isnt the best way to fetch a chunk of documents? When we subscribe meteor creates observer to watch db changes. Also it uses merge-box mechanism on client and server and it could be very expensive for production

Well, it’s nice to have reactivity, and I could actually switch it to using method calls by toggling a single boolean :slight_smile:

My own app is a semi-internal business app that is never going to have too many concurrent users on the same instance, so server load isn’t a big deal for me. But yeah I’ve heard people who make public apps often end up switching to nonreactive methods.

Hmm… A neat optimization could be to switch from sub to method calls as soon as the query changes (when the user scrolls or does a search), and then if the query doesn’t change for 5 or 10 seconds or something, you switch to a subscription again. Maybe something worth adding to my grapher-vue package :thinking:

So, while I’m able to get this solution to work somewhat decently - the issue that it is not reactive. Is there an example for an efficient infinite scrolling list which is reactive (using react)? Or, if all I really need is reactivity on user interaction (such as a like), how can I include that in? I wonder if this is sort of killing the purpose of using meteor, if i’m not using the reactivity that’s supposed to be built in.

edit: to be clear, is there some sort of reactive, virtual list, react component setup for meteor? Something that ideally will only render the nodes in view, and keep a subscription of only the nodes in view as well, so that it can be reactive and efficient?

Thanks

Hello, I can’t vue js subscription re-call change by click event. But first time usually initialised. I want to re-subscription change by event. Please help me.