How to have reactive total counts on a publication with pagination?

Hello, I was trying to add pagination to one collection I have but now the counters that I used with different selectors are broken due only getting certain amount of docs and not all in a go, our frontend is using blaze and we don’t want to add any atmosphere package as most of them have stopped updates in a while, so I’m pretty much stuck trying to figure out how to get the counters to be reactive again without loading all the docs.

I tried using a Method call to get all this but seems the helper doesn’t like it. As I wrapped it on a promise to wait for it to be resolved but I always get [object Promise] rendered

Any Idea on how to achieve this without any 3rd dependency?

Probably easier if you publish the count. If you don’t want to add a dependency, you can simply look at the code in tmeasday:publish-counts (it’s very straightfoward) and roll your own. This code might not scale well though so there’s an alternative suggested: natestrauser:publish-performant-counts (but I’ve not tried that one myself).

1 Like

Store the count in a separate collection and subscribe to that collection.

2 Likes

I solved this in several projects by having a special collection called connection that gets automatically subscribed to, based on the DDP connection id. When publishing a subscription I call a special function that takes the query and strips the options (step, limit) and then updates the connection with the name of the publication and the max count.

I know, this seems like a lot of work, but I tried all different sorts of approaches to this issue, and wanted to come up with a pattern that works no matter where I use it.

3 Likes

Sounds really interesting @jamgold, do you have a quick example? trying to wrap my head around, how you achieve stripping down those step/limit options using that function you commented.

Thanks for sharing!

Our count is specific per user and having a new collection with thousands of count that is then updated (eg. every 10 seconds like in the solution of Nate Strauser’s package) isn’t feasible.

We have CRON like jobs that run bulk updates on the users data and will then force a recalculation of these counts on our backend server despite the user not even logged in and there it not being necessary.

We’ve looked at these packages and they don’t seem like a good solution for our specific app but thanks for the comments!

@mexin thanks for the interest. Here is a quick working approach.

First we define a new collection on server and client. It will contain the maxCount for a publication, per connection

Meteor.collection_count = new Meteor.Collection("collection_count");

On the client we subscribe to the collection-count publication, doesn’t matter if logged in or not, because it is per connection

  Meteor.subscribe('collection_count');

On server side we add the following

  // publish the collection_count per connection id
  Meteor.publish('collection_count', function () {
    const self = this;
    return Meteor.collection_count.find({ connectionId: self.connection.id });
  });
  // make sure we clean up when the connection drops
  Meteor.onConnection(function (connection) {
    const connectionId = connection.id;
    connection.onClose(function () {
      let c = Meteor.collection_count.remove({ connectionId: connectionId });
      if (Meteor.isDevelopment) console.log(`onClose ${connectionId} removed ${c}`);
    });
  });
  Meteor.startup(function () {
    //
    // clear collection_count when server starts
    //
    Meteor.collection_count.remove({});
    Meteor.collection_count._ensureIndex({ connectionId: 1 });
  });
  // add function to do the collection count
  Meteor.collectionCount = (subscription, cursor) => {
    check(subscription?.constructor?.name, 'Subscription');
    check(cursor?.constructor?.name, 'Cursor');
    const connectionId = subscription.connection.id;
    const subscriptionName = subscription._name;
    const cursorDescription = cursor._cursorDescription;
    const collectionName = cursorDescription.collectionName;
    const db = cursor._mongo.db;
    const collections = db.collections().await();
    const collection = collections.filter((c) => {return c.s.name == collectionName});
    const maxCount = collection.length == 1 ? collection[0].find(cursorDescription.selector).count().await() : -1;
    console.log(`Meteor.CollectionCount ${subscriptionName} ${connectionId}`, maxCount);
    Meteor.collection_count.upsert({ connectionId: connectionId, subscriptionName: subscriptionName }, { connectionId: connectionId, subscriptionName: subscriptionName, maxCount: maxCount })
    return cursor;
  }

Now we can use this function in our publications

Meteor.publish('items', function(query={}, skip=0){
  const self = this;
  return Meteor.collectionCount(self, Items.find(query, {
     limit:10,
     skip: skip,
     sort:{number:1}
  }));
});

On the client we can simply query this collection

Template.items.onCreated(function(){
  const instance = this;
  instance.skip = 0
  instance.subscribe('items', instance.skip);
  
  instance.autorun(function(){
    const c = Meteor.collection_count.findOne({ subscriptionName: 'items' });
    console.log(`items maxCount = ${c?.maxCount}`);
  })
});

I’m pretty sure this can be improved upon by someone with better coding skills.

Edit: I just created a new package meteor show jamgold:meteor-collection-count

2 Likes

None of the answers do what you really want becase none provide solution that is reactive.

Hi @judit666h, welcome to the forums!

Can you expand on your comment? Did you try out these methods and have issues with them?

From my experience, all of these methods are reactive. If they aren’t for you it’s likely an issue with making sure you are fetching the data in a Tracker.autorun or blaze helper

You could use MongoDB aggregation to achieve this, but in order to get a reactive solution, you will need a 3rd party package. (It won’t affect your client bundle size).

import { ReactiveAggregate } from 'meteor/tunguska:reactive-aggregate';

On the server

Meteor.publish('scoreCount', function() {

let pipeline =   [
    {
      $match: {
        score: {
          $gt: 80
        }
      }
    },
    {
      $count: "passing_scores"
    }
  ]
  ReactiveAggregate(sub, collection, pipeline, {clientCollection: 'scoreCount'} );
});

On client

const scoreCount = new Mongo.Collection('scoreCount');

Meteor.subscribe(scoreCount)

let scoreCount = scoreCount.find().fetch() // Get the score 
1 Like

Just reminder, findOne on the client is not reactive, find is : )

This is not correct, both are reactive.
In fact, findOne is an alias for find(selector, { ...options, limit:1 }).fetch()[0]

1 Like

I have tried it on the client in my application, and it doesn’t work :thinking:

Maybe I’m doing something wrong, can you verify when you have a chance?

Can you start a new thread with enough code to replicate the issue?
It’s likely something in the surrounding context, likely it’s not being run in a reactive function / computation

Will do, thanks for referencing the source code, you are right, it’s clearly an alias, I will give it another go.

here’s what I always do:
I create a method to load the data, let say users. You can return counters, joints with other collection, aggregations, whatever you want.

Then you make a subscription to the the same user collection, but you keep the subscription extremely small, example.

Meteor.user.find(
  {updatedAt: 
    { $gte: new Date() }
  }, 
  { fields: 
     { _id: 1 }
  }
)

Then in your autorun, every time your user collection changes via the subscription, you refetch the data using your method.

This approach has several advantages:

  • You keep memory load low (publications have quite some overhead)
  • As explained, you can do much more complicated things without having to hack the publication mechanism or using weird packages that do so.
  • You can also subscribe to changes on joint or aggregated data, and then refetch the data via the method; reactive joints.
  • It’s still reactive

Publications are a cool feature of Meteor, however, but kind of useless once you need to do more complex things with your data. Even for things as simple as counters.

1 Like