How to efficiently implement endless scrolling?

I would like to implement something like Facebook’s notification list, which shows you all notifications you’ve ever received. To limit the number of initially displayed notifications, it uses endless scrolling to load additional (older) notifications on demand as you scroll down.

I know how to limit the initial result set and extending it as the user scrolls down (using limit), that’s clear. But: this list of notifications also references the users that have sent the notifications. I would like to show their names and avatars, just like Facebook does.

For such reactive database joins, I’ve used reywood:publish-composite so far. The straightfoward approach with this package would be to use limit on the parent collection (i.e. the notifcations) and get the user associated with a certain notification in a child query.

The problem I see here is performance. If the list grows longer and longer (because I increase the limit), and for every notification an additional query (for the user) would be made on the database, this would result in a lot of database queries. And, even worse, some of these queries would not be necessary, because the same user is related to more than one notification.

The other option would be to collect all notifications in one subscription, collect all unique user ids contained in these notifications in an array and then run a second subscription based on this array.

This would reduce the number of search queries to 2, but still the subscriptions would run every time I increase the limit. Is Meteor clever enough to detect this and only send the “new” documents over the wire in this case? Or is there another approach (maybe using the low-level pub/sub API) to implement endless scrolling in an more efficient way?

1 Like

I’d like to do this for my lists too. I looked into it, but didn’t want to waist too much time, so I ended up implementing a ‘load more’ button. I think the only thing left I have to do is replace the ‘load more’ button with an event on the window scroll.

Wouldn’t you just get a batch of say, 10 at a time?

I’m incorporating a “notifications” view to almost all my apps. For line-of-business type apps, it helps to show the user what their colleagues have been doing with the “shared data” so that people can collaborate easily.

One pattern I’ve come to accept and apply is creating a denormalized notifications collection.

Any data or action that needs to end up at the user as a notification gets transformed into a notifications collection where there are at least:

  • who this notification is for
  • the count of such actions (ie the same data has seen multiple updates, so one notification with incremented counts)
  • a short excerpt to be used as notification text
  • a link to the most appropriate place in the app that shows details about this notification
  • who (name or name & lastname) this notifcation is from (like your usecase, the last - because I’m combining multiple actions into single notification as explained above - user who has caused this notification)
  • avatar of the said user
  • date of last action on this notification bundle

I also generally send out notifications-summary emails every once in a while, depending on the application’s usecases this can be from a few emails every day to one per week

And I finally run a background task that cleans up unread notifications that are more than some time old (generally a month or so)

This has worked quite well for me and it can be built upon depending on your application’s notifications velocity.

1 Like

Thanks for your detailed reply. Yes, denormalization would be an option. But the downside is that the notification reader won’t get the most recent info about the user in this case. For instance, if a user changes the avatar later, older notifications would still refer to the old one.

You could of course update all collections where you’ve referenced the avatar once the user changes it, but this would also result in a lot of database updates.

So I would prefer to do this in a more “normalized” way, at least for my app where changes in the user info are quite likely.

Will you elaborate on this more? I too get asked for more “visible” notifications. Right now, I have a badge next to buttons or navigation lists that indicate how many of an item there are “new”. Once they’ve view whatever list that is, the badge goes to zero. But my clients still say the users need more visibility – some have asked for big red text or similar.

I do the same. Due to joins lacking reactivity without a lot of hacking (I’ve done it, but it’s ugly), I usually denormalize. It’s a different mindset from SQL, where normalization is encouraged.

Well it is not just a SQL/noSQL thing.

Denormalization is a common design pattern that predates noSQL databases. In fact, any kind of “summary” view is a good candidate to be denormalized on any data backend.


some have asked for big red text or similar.

I use a combination of

  • dynamic favico to show total notification count,
  • sound alert whenever notiifcation count increments
  • browser notification
  • animated notification icon in a prominent place on the ui
  • email reminders

You could of course update all collections where you’ve referenced the avatar once the user changes it, but this would also result in a lot of database updates.

@waldgeist you’d be surprized how infrequent such updates would be. Besides, if you “cap” the number of notifications a user can have - and contextually, notifications should represent “recent” actions anyway - updates on the normalized fields won’t be too much of a burden on your app.

also it does not feel like u would need reactivity on both, so little low level publish with observer above 1 collection and doing fetch from user collection does not sounds like performance killer.

This is where I was going too. I tried Buzz for the sound, but got errors on compile in 1.0.3. What are you using for sound? What package are you using for browser notification (this is only for Chrome right)? I saw font awesome had animated icons, but none looked the part.

Buzz works for me fine on 1.2.1 I never had any blocking problem with it on previous versions, either.

For notifications (supported browsers) I use notifyjs I think there is a wrapper on atmosphere, I just use the original one.

For animated icons, I use animatecss and actually copy over the “relevant” pieces of css to my project for the animations I use. You can apply this on any icon library.

1 Like