Dynamic aggregations with SparrowQL

announce
1

Hi all!

If you are interested in something like Grapher, but for dynamic aggregations, here it is: SparrowQL. It’s more about Mongo itself, but it really helped us in few Meteor apps. Hope you like it!

9 Likes
2

Hi @radekmie_can you include in your blog or here perhaps how to install SparrowQL in our meteor application?_

3

I’ll try to write something more about it, but here’s a short how-to:

const PostsCollection = new Mongo.Collection('posts');
const Posts = PostsCollection.rawCollection();

// Use Posts as in the article.

And that’s all. In Meteor, all collection have their bare, MongoDB driver collections available - just use rawCollection to get them. Just keep in mind, that these are all completely asynchronous, and you have to await them!

4

The reason why Grapher hasn’t used the $lookup approach is because:

  1. Does not work on sharded collections, which is silly
  2. There is proof that using lookup and nested lookup is extremely unperformant (https://github.com/cult-of-coders/grapher/issues/188)
5
  1. Yeah, it can be a problem, but as long as you are not using sharding - it’s cool.
  2. SparrowQL is not doing nested lookups: it’s a flat pipeline, with many $lookup phases (it was designed to work with Mongo 3.2). To be honest, I haven’t noticed any performance problems, even for 6 lookups from large collections (it’s always a problem about what is large, these were >1m records), as long as we had correctly indexed fields. What is really cool, is the streaming: you can iterate over the aggregation cursor. It was the main reason, as we at first used it for reporting (i.e. streaming a ton of joined and projected data either to S3 or another collection).
1 Like
6

@radekmie i started playing your package and encountered some challenges in publishing it to the client

const relations = [
  { to: "Customers", from: "Orders", foreign: "_id", local: "customer_id" }
];
const projection = {
  ordersCustomer: "Orders.customer_id",
  firstname: "Customers.firstname",
  middleName: "Customers.middlename"
};
const start = "Orders";
const pipeline = build({ projection, relations, start });

Meteor.publish("CustomersAndOrders",async function() {
  let self = this;
  let cursor = await Orders.rawCollection()
    .aggregate(pipeline)
    .toArray();

???  what's next?
  return cursor; 
});

Thanks for the hep

7

There’s no way to publish an aggregation cursor. You can either do it “one time” with low-level publish API or use it in methods, not aggregations.

8

Got the solution…

Meteor.publish("CustomersAndOrders", function() {
  const relations = [
    { to: "Customers", from: "Orders", foreign: "_id", local: "customer_id" }
  ];
  const projection = {
    ordersCustomer: "Orders.customer_id",
    firstname: "Customers.firstname",
    middleName: "Customers.middlename"
  };
  const start = "Orders";
  const pipeline = build({ projection, relations, start });

  ReactiveAggregate(this, Orders, pipeline);
});

I just missed that build functionality is to build the aggregation pipeline… @radekmie i really like this package! Thanks for the effort :heart_eyes:

1 Like