Aggregation to sample down a data collection

help
1

I need to graph some data, the problem is that the data quantity is quite big and I’ve been doing some stress tests, and sometimes the web client hangs due to the huge amount of data (quantity that I expect to have from real data)

So I made this, looking at some solutions in the web

db.rtphistory.aggregate(
  {$match: {topicId: "yXNz7p2cif4H4ktgz"}},
  {$project: {value: 1, _id: 1, createdAt:1, created: {$trunc: [{$divide: ["$createdAt", 1000]}]} }},
  {$group: {
    _id: "$created",
    value: {$avg: "$value"},
    createdAt: {$first: "$createdAt"},
    myCount: { $sum: 1 },
    }
  },
  {$sort: {"created": 1}}
)

Being the “1000” the chunks of data I’m reducing to an average (it will be a variable, but for now I’m fixing it to 1000), and createdAt is a timestamp, in other words, I’m making chunks of data of 1 second long, I might change it to longer frametimes.

(I put a count there to know how many points of data I’m reducing to 1 average, pretty much for debugging this solution, it will be gone later)

The idea is that on the client-side, the quantity of data to handle is reduced.

How can I make this work on Meteor publish?
I’ve been reading but can’t wrap my head around the way you use aggregates in Meteor.

Please let me know if you need more info.

EDIT: also, maybe there’s a simplier way to do this on Meteor and I’m just missing something?

2

To use aggregate inside a publication your best bet is to check out this package:

By the venerable @robfallows, so should be of excellent quality. Some additional info here: Tunguska:reactive-aggregate

1 Like
3

he guided me once to a solution before, and now he keeps helping me on my missventures hahaha
Thanks @vooteles , gonna give it a try

If I solve this, I’m comming back to post the method and results

1 Like