How to group items from cursor without fetch?


#1

Hey, Meteors!

Intro. Let say we have Messages collection. And every doc in that collection looks like this:

{
  createdAt: 1432018583531,
  message: 'Hello',
  username: 'chuk',
  chatId: '82rWv8bv4YbxEiCky'
}

It’s easy to render plain list of messages:

<template name="messages">
  {{#each messages}}
    {{> singleMessage}}
  {{/each}}
</template>
---
Template.messages.helpers({
  messages: function () {
    return Messages.find(selector, options);
  }
});

Output may look like so:

  • @chuk [May 18, 10:00 AM]: Hello
  • @geck [May 18, 10:01 AM]: Hi
  • @geck [May 18, 10:01 AM]: How R U?
  • @chuk [May 18, 10:02 AM]: Fine :slight_smile:
  • @chuk [May 19, 09:02 PM]: Hey?


Question. How to group messages by date and by author? How to transform output to something like this:

May 18

  • @chuk, 10:00 AM
    — Hello
  • @geck, 10:01 AM
    — Hi
    — How R U?
  • @chuk, 10:02 AM
    — Fine :slight_smile:


May 19

  • @chuk, 09:02 PM
    — Hey?


Thoughts. The first idea is to use .fetch() and group messages imperatively by iterating them all:

<template name="messages">
  {{#each dateBlock}}
    {{#each messageBlock}}
      {{#each}}
        {{> singleMessage}}
      {{/each}}
    {{/each}}
  {{/each}}
</template>
---
Template.messages.helpers({
  dateBlock: function () {
    return makeDateBlocks(Messages.find(selector, options).fetch());
  },
  messageBlock: function () {
    return makeMessageBlocks(this);
  }
});

But I believe that’s a bad way, because .fetch() will cause recomputing all the data on every change. The growing list will slow down depending on the number of elements.

What is the best way to do such grouping? Hey, meteors? May be we need to insert delimiters and give special classes to the elements via jQuery depending on data-attributes in afterFlush callback? Or to use cursor.observe and fill up special local collection? Or?


#2

Sounds like you want a sort specifier http://docs.meteor.com/#/full/sortspecifiers

Something like Collection.find( {}, { sort: { datecreated: 1 } } )


#3

If sort specifies don’t work you should look into the mongo aggregation pipeline.


#4

No. I don’t have problem with sort, I use {sort: {createdAt: 1}}, of course. I need a non-existent group specifier :slight_smile:


#5

If you wanted to sort the messages by date, but have already sorted the messages by date then im not sure what your asking.


#6

I want to group already sorted messages.


#7

Ah I think i see what your looking for. So instead of every message coming out as:

[10:01]bob: hello im bob
[10:01]bob: whatsup

You want to group messages that come in the same minute together:

[10:01] bob:
-hello im bob
-whatsup

correct?

If so, what @khamoud suggested seems like it would work: http://docs.mongodb.org/manual/core/aggregation-pipeline/

Otherwise, the basic logic would just be to get the timestamp, and if a message block for that time hasnt been created, create a new messageblock with the messages, else append the message to a matching messageblock.

Edit: Also there seems to be a few chat packages on atmosphere…check them out.


#8

I try to find the best and fast solution to create that message blocks. I used to use .fetch() for that and imperative approach, but I’m not happy with that, because it breaks fine-grained reactivity and recomputing the whole list on every change, that’s not good for chat app.


#9

Related but not exactly

I had to deal with something similar in my app, but could not use the aggregation pipeline since the results were coming out of two different publications of the same collection, and I had to group my data by a common ID.

This is how I worked around it: I subscribed to both publications, and once done I created a helper that minimongoes the published data into groups.

<template name="Rows">
    {{#each rows}}
       <div>Row {{this._id}}</div>
       {{#if grouppedDataMatches this}}
       {{ > groupedDataTemplate this }}
       {{/if}}
    {{/each rows}}
</template>

<template name="groupedDataTemplate">
    {{! Show your groupped data in any way you want }}
</template>

----


Handlebars.registerHelper('rows', function(){
   return RowsCollection.find({query that gives results with unique timestamp or whatever});
});

Handlebars.registerHelper('grouppedDataMatches ', function(parent_row){
   return RowsCollection.find({query that gives results related to parent_row});
});

Of course this can’t work with everything, but seems to be working fine and fast for me.


#10

@artpolikarpov is your performance concern an actual observation or purely theoretical ?
Many time, performances concerns exist only in the mind of the developer and doesn’t translate in actual problem in practice. Morever with a client app, computations are done on the client’s computer. :wink:

Now, if this is really slow, have you tried cursor.map instead of fetching then working on the resultant array ?
The doc says :
> When called from a reactive computation, map registers dependencies on

the matching documents.

I have not made tests, but perhaps it could improve your situation, and lead to fewer reprocessing.


#11

That is an actual observation. The bigger list is slower list with fetch :frowning: Directly proportional relationship.


#12

I’m curious how others fixed this as well. Grouping data from multiple (compositely) published collections without having to re-run a huge group function on every change.

FWIW, I use one function that get’s called on every reactivity change and group that in a flat array/object structure for every level I want to group on. Hacky, but it works. Not very re-usable yet though.


#13

Why so complicated? It appears to me that a simple brilliant application of list (and map/object) comprehension calls such as available from underscore or lodash would easily and quickly solve the problem at hand.

The processing, in order, would be… (assuming they’re already sorted by createdAt; also assuming lodash as _ since I’m not as familiar with the underscore library)

  • “round” all the createdAt dates to the minute (using _.map)
  • group all messages by createdAt (using _.groupBy)
  • remove the createdAt property for all but the first item (using _.each on the list of groups, then _.map again on the grouped-together messages, the latter removing the property when the index > 0)
  • then flatten that out into an array again (using _.values, then _.flatten)
  • return it as the model for the template to work on

The template then simply checks for existence of createdAt in the current item and if it finds it then it uses the output format with date, and if it doesn’t it just outputs the message itself. Should work mostly. It might be that additional trickery then in Template.messages.onRendered is required, running through the list using JS/jQuery once more, but I’m not sure about that, and it would be minor – like, I imagine, adjusting the parent/child relationship / grouping up all message-without-date-messages in a common container or so, which you cannot do with just the handlebars template. (I don’t want to say “cannot” because there’s always some way to do things… ;))

Does that help? I think that should solve the problem without any excessive .fetch()ing etc, just pure client-side grouping and minorly adjusting the publication results.

EDIT: And sure, you could move into MongoDB aggregation territory. But generally, whatever you can do purely on the client, you should. And here, using MongoDB aggregation would mean losing the power of reactive subscriptions, so it’s not something to do lightly.


#14

FYI. Meteor accepted my feature request «Grouping data from cursors reactively»: https://github.com/meteor/meteor/issues/4408