Meteor Aggregate not showing correct results

[shabbiralsaba]

I have added this package like below

meteor add meteorhacks:aggregate

and my code is like below

Favorites.aggregate([
        {
            $lookup: {
                from: "products", 
                localField: "product_id", 
                foreignField: "_id", 
                as: "products"
            }
        }
    ]);

but result in something like that

AggregationCursor {
   pool: null,
   server: null,
   disconnectHandler:
    Store {
      s: { storedOps: [], storeOptions: [Object], topology: [Object] },
      length: [Getter] },
...............

but it’s working on shell command.

What can I do now?

Thanks

[robfallows]

The MongoDB library has changed the way aggregations are returned and the meteorhacks:aggregate package is unmaintained, so has not kept pace.

However, the AggregationCursor object has a toArray() method, which returns a Promise to the result you’re expecting. You should be able to use something like:

import { Promise } from 'meteor/promise';
// ...
const results = Promise.await(Favorites.aggregate(...).toArray());

In fact, you can do all this without using meteorhacks:aggregate and get more portable code as a result.

[shabbiralsaba]

Thanks @robfallows,

and now showing result like that

[ 
    { 
        _id: '5ddDcgeYZBhRuQivD',
        product_id: 'Hwrq4yYLGTpFtz9g7',
        products: [ [Object] ] 
    } 
]

and the question is how I extract the products object

products: [ [Object] ] 

Thanks

[robfallows]

That’s just what vanilla console.log does with nested objects. Try:

console.log(JSON.stringify(result, null, 2));

[shabbiralsaba]

Ohhh! very nice,

Thanks, @robfallows

[jamgold]

In fact, you can do all this without using meteorhacks:aggregate and get more portable code as a result.

do you happen to have a link handy that explains how to do that without the meteor hacks:aggregate package?

[tomsp]

How to convert `Promise` to `Array` in `Aggregation of rawCollection`?

Here’s one way: import { Promise } from 'meteor/promise'; const data = Promise.await(Branches.rawCollection() .aggregate([ { $match: { _id: '001' }, }, { $project: { _id: 1, name: 1, createdAt: 1, createdAtText: { $dateToString: { format: '%Y-%m-%d', date: '$createdAt' }, }, }, }, ]) .toArray());

[tunifight]

Thanks for the Promise.await hint. I was not aware that Meteor uses this syntax to avoid async await and keep fibers like sync syntax

[robfallows]

do you happen to have a link handy that explains how to do that without the meteor hacks:aggregate package?

Not really. Although, to be fair, I’ve not looked!

As with all these things, portability comes with a price. In the case of Meteor, that price is increased verbosity and more upfront work. Depending on your appetite for that, you could consider making incremental changes towards portability and stopping when the returns diminish too far.

Fortunately, you can make a huge step just by removing meteorhacks:aggregate and using the underlying MongoDB methods, some of which Meteor wraps for you. The meteorhacks:aggregate package just adds another method to Meteor’s Mongo.Collection object. It adds the underlying MongoDB library aggregate method, wrapping its callback in Meteor.bindEnvironment to make sure it’s Fiber-based. In principle, that’s fine - usage appears consistent and portable. Unfortunately, in the case of aggregate, the type of result was changed from an array to an aggregation cursor, which the package (being unmaintained) does not support.

Let’s just discuss Fibers for a moment. I love them - they make coding so simple when they’re managed invisibly, as Meteor does for the most part. However, they’re not widely used in the vast majority of JavaScript projects. Also, they’re not available for the browser, since the code has to be built for the underlying OS, with all the various dependencies that needs. For portability today, that means using callbacks or Promises.

Let’s talk about callbacks. Not for long, because they’re horrible, but they are the JavaScript “old school” way of handling asynchronicity. As such, they’re supported everywhere. However, using callbacks in Meteor server code often gets you a “code must always run within a Fiber” error. The consequence of that is you need to wrap the callback in Meteor.bindEnvironment or wrap the method in Meteor.wrapAsync. Neither is particularly portable.

Onto Promises. You’ll probably already know I’m a fan - Promises were a stepping stone towards a coding style gaining widespread adoption. Of course, I’m referring to coding using async / await. In many ways this brings Meteor’s groundbreaking sync-style of coding to JavaScript coders everywhere.

What’s this got to do with the MongoDB aggregate method? Well, some time back all asynchronous MongoDB methods were refactored to support Promises. (They also continue to support callbacks - but let’s not go there.) Since they return Promises, we can adopt modern JavaScript coding styles - particularly async / await to get code which is more portable very easily. Note, I did not say completely portable.

Portability 1

1. Remove meteorhacks:aggregate and any imports.
2. Ensure you have import { Promise } from 'meteor/promise'; in code using aggregation.
3. Change result = myCollection.aggregate(...); to result = Promise.await(myCollection.rawCollection().aggregate(...).toArray());

As you will see, that’s not especially portable. However, it does get rid of meteorhacks:aggregate. It relies on Meteor’s Promise package, which implements the (non-standard) Promise.await() method and uses Fibers to hide its asynchronous nature. It also uses rawCollection().

Portability 2

This assumes you’re using a Meteor method to do the aggregation. I know Meteor methods are not portable, but this is an easy way to introduce async / await.

1. Remove import { Promise } from 'meteor/promise'; unless you’re using explicit Promises somewhere else.

2. Change the method declaration to async. So, for example:

   Meteor.methods({
     myMethod() {
       // ...
       return result;
     },
   });
   

   becomes:

   Meteor.methods({
     async myMethod() {
       // ...
       return result;
     },
   });
   

3. Change result = Promise.await(myCollection.rawCollection().aggregate(...).toArray()); to result = await myCollection.rawCollection().aggregate(...).toArray();

That’s much better - we now only have rawCollection() as a non-portable element.

Portability 3

We could choose to hide rawCollection().aggregate by defining an aggregate method on the collection (similar to meteorhacks:aggregate). That would yield portable syntax for using the aggregate method, when coupled with async / await. Something like:

myCollection.aggregate = myCollection.rawCollection().aggregate.bind(myCollection.rawCollection());

If you have a single importable file for the collection, that would be a good place.

Portability 4 and on

This is where it’s beyond the scope of a forum reply and beyond my willingness to go further .

[jamgold]

Thank you very much for taking the time to respond. Since I only use it server-side in one place I went with P1. One little typo, it needs to be

import { Promise } from 'meteor/promise'; 

[robfallows]

One little typo

Corrected - thank you for spotting that

[illustreets]

An excellent, informative post!

Edit: I was working on a strategy for exactly that just recently.

[jamgold]

As much as I am glad the solution from Portability 1 works, I am trying to solve it with Portability 2, but can’t get it to work (probably because I don’t understand this async stuff yet).

I defined an async function

const aggregate = async function(collection,aggregation) {
  var result = await collection.rawCollection().aggregate(aggregation).toArray();
  console.log('aggregate result',result);
  return result;
};

which prints out the correct result, but when I call the function

const resultArray = aggregate(MyCollection, theAggregation);

resultArray is a Promise, and I don’t understand why. I am actually trying to use the result of the aggregation in a publication, but can’t get it to work.

[tomsp]

An async function always returns a promise. Check the specs.

So you can do this

aggregate(collection, aggregation).then(console.log).catch(console.error);

You could also try do make your publication an async function and await your aggregation, but i never tried that. With methods it would work.

[jamgold]

Thank you. I simply had to turn it into a method and then was able to call it synchronously in my publication, something like this

Meteor.methods({
 aggregate(){
    return aggregate(Collection, myAggregation);
 }
});
Meteor.publish('publication', function(){
 const ag = Meteor.call('aggregate');
 return [];
});

[tomsp]

That is quite bad though bc anyone can call this method. You should only ever add meteor methods for client/server communication.

That being said, what you can try is this

Meteor.publish('publication', function(){
 const ag = aggregate(Collection, myAggregation).await();
 // or
 const ag = Promise.await(aggregate(Collection, myAggregation));
});

[jamgold]

You are totally correct. The missing piece was the await(). The following works in my publication

 const aggregation_result = aggregate(Collection, myAggregation).await();

[ggerber]

Remove import { Promise } from 'meteor/promise'; unless you’re using explicit Promises somewhere else.

Should one therefore explicitly use

import { Promise } from 'meteor/promise';

if one uses Promise.all() somewhere else?

[robfallows]

If you’re using Promises in that file, then you’ll need to include the import in that file. Otherwise, it just adds “noise” to the code.

You don’t need to do the import if all you’re using is async / await.

[jam]

thanks @tomsp, this was helpful. out of curiosity, where is .await() coming from here? Is that built into Meteor?