Aggregation and publish


#1

I have a collection of locations, where I insert data like this:

{
    name: name,
    description: description,
    loc: {
        'type': 'Point',
        'coordinates': [
            longitude,
            latitude
        ]
    },
    owner: this.userId
}

I’ve added the 2dsphere index on the collection:

db.locations.ensureIndex({'loc':'2dsphere'});

Using aggregation (with meteorhacks:aggregate), I can calculate the distance to the locatons, like this:

Locations.aggregate([
    {
        $geoNear: {
            near: {type: 'Point', coordinates: [longitude, latitude]},
            distanceField: 'distance',
            maxDistance: distance,
            spherical: true
        }
    }
])

How would I use the aggregation with publish? The aggregate method returns an array, and not a cursor.


#2

Add the results of the aggregation into a new collection and publish that. When you rerun the aggregation use update/upsert and remove invalid entries instead of clearing the whole collection and adding the results back in if possible - so that only the changes are pushed to the client not the whole collection again.


#3

You dont have to publish a cursor. You can just use your array data in the publish function and use ‘this.added’, ‘this.removed’ and ‘this.changed’. Have a look at the ‘counts-by-room’ example on http://docs.meteor.com/#/full/meteor_publish


#4

Thanks! I ended up with a publish method like this:

Meteor.publish('nearest-locations', function (latitude, longitude, radius) {
    var self = this;
    var handle = Locations.aggregate([
        {
            $geoNear: {
                near: {type: 'Point', coordinates: [longitude, latitude]},
                distanceField: 'distance',
                maxDistance: radius,
                spherical: true,
                sort: -1
            }
        }
    ]).forEach(function(location) {
        self.added('nearest-locations', location._id, {
            name: location.name,
            latitude: location.loc.coordinates[1],
            longitude: location.loc.coordinates[0],
            distance: Math.round((location.distance / 1000.0) * 10) / 10
        });
    });
});

I’m not sure how I should implement the this.removed and this.changed methods in my publish method…

Edit: changed the name of the publication, for clarity.


#5

Hi werner,

As you probably know the aggregation function isn’t reactive so after the first run you wont receive any more data from the db. Unless you plan to add more documents to the client some other way you might as well use a method. On the client you can have a client only collection where you insert the result from the method.

So why not use a publication? Because when you do the server keeps a copy of every document you published in memory. Again, if you don’t plan on publishing more documents there is no reason to do so.

Hope that helps!


#6

Hi Werner,

I agree with Carl. If you just want the nearest locations at that time use a method. If you want to use the method results in a template see http://stackoverflow.com/questions/22147813/how-to-use-meteor-methods-inside-of-a-template-helper . If you want the ‘nearest-locations’ to update automatically with a change in the ‘Locations’ collection use a publish with an observe or have a results collection thats kept up to date with observe or methods or something. For example if you or another user added a new location to the collection or a location changed. It depends on you end goal.

To get the aggregation in the publish reactive use a observe in the publish. Exactly like in the ‘counts-by-room’ example in the meteor docs. With the observe you know if a new location has been added, changes or removed. For simplicity re-run the aggregation each time (except remove) and if the location was previously published then update the publish ( http://docs.meteor.com/#/full/publish_changed ) , if the location was removed then remove the location from the publish ( http://docs.meteor.com/#/full/publish_removed ) and then for a new location use ( http://docs.meteor.com/#/full/publish_added ).

Some notes on this:

  • Follow the ‘counts-by-room’ example to only start observing after you initialized the publish when a user subscribed. And remember to stop the observe once the client unsubscribed.
  • Use specific queries to only observe what is needed. Like just for the subscribed owner.
  • Use observeChanges insdead of just observe if possible. Its more efficient and think it provides better mongo oplog support.
  • If possible do not use aggregations on every change as it can be a resource intensive task. Maybe just for the initial then use math to calculate the change in lat, long, distance if the location added/changed or something like that :smile:

We developed a stock forecasting and a prediction system for complex fertilizer mixtures that used aggregations over lots of transactions and we quickly realized that its to slow to provide instant feedback if a user change something in our graphs and tables. So we ended up not using it at all. We used a few results collections that is kept up to date with a observe on the transactions collection. So if a transaction is added or changed we just do the math for the predictions (and other stuff) and determine the influence of the change and we either increment or decrement the necessary values in the results collections. Dont know if it will be possible in your scenario.

Regards,
Riaan


Why is mongo aggregation not reactive in meteor?
#7

Hi!

A lot to dig into here. Thanks! :slight_smile:


#8

Hi Werner,

you can publish all your locations/depends on local position and find nearest on the client side using only minimongo, without server


#9

@werner you might take a look at this example that shows how to use ‘observeChanges’ in order to get the removed, changed, and added functionality that you are talking about implementing in your example.


#10

Just an idea about using $out operator of the aggregation pipeline:

https://docs.mongodb.org/manual/reference/operator/aggregation/out/

You could use this to build a new collection and publish and subscribe to this one.

Maybe this is also a kind of solution.


#11

Hi all,

I made an NPM package to help publish aggregated values: meteor-publish-join. Hope it will help


#12

Here is a atmosphere package I’ve published recently that may help publish-aggregation


#13

This is awesome! Nice work @nlhuykhang