I am about to start writing an application which will have as the main feature a long, infinite scroll powered Items List based on the categories parameters an user choose to get the list.
The List Items can be static. They don’t need to be updated. When I choose an Item, then I would like to subscribe to a specific Item. But it feels not really naturally when I think about the List and the subscriptions chaos while scrolling and clicking on different categories… And I don’t like to think about the wasted resources on the server side, because each subscription will trigger a lot of tasks on the server for nothing.
So, what’s the best solution for this, if I want to use Minimongo anyway on the client? Should I just get the data with a Meteor.call to a method and store the data in a client only Collection or are there some better solutions out there for this kind of stuff?
Do you think it makes sense to cache the running subscriptions? If I understand the subs-manager package right, it just don’t stop the running subscriptions if I change my options. It means I would produce a lot of memory waste on the server side, or not?
I suggest you read the articles listed on the github page. @arunoda will be able to provide a better answer than I can, but I think he will also refer you to his articles, which talk about it in more detail.
Thank you @streemo. It’s nice that you want to help, but yes, I already read these articles
Actually I wanted to hear from someone who had the same headache and came up with a solution, which satisfies.
If your data set if the same and does not change much, then using a subscription is much better than using a method. That’s because Meteor caches data on the server for similar queries. That reduces your network, cpu and mongodb(load) cost.
But, even that could be an issue. So, the best option is to generate some static JSON files based on your data. Then put it them to a CDN or a file server. Then pull those via AJAX and render it to the screen. You can also fill that fetched data into local mini-mongo collection.
(You can also generate those JSON data as a REST apis as well. Then use something like Cloudflare or Varnish to cache data dynamically)
UI Issue
This will be the hardest part. Try to pay attention to that. @chandika could add more info on this.
@arunoda at what point do you recommend using a CDN to serve posts to the user? This is something that should be done at scale only?
I have posts which I want to send to the user 20 at a time, paginated. Each user can ask for the next 20, etc. and the entire list could easily be 1000 posts. But I want them to be able to sort by top, recent, etc. So, currently I am caching every subscription since the user will likely go back and forth between sort options. In what way would a CDN help here?
@arunoda, thank you to join the discussion. the approach with the db in the cloud is very interesting. I guess we could generate periodically a file to to store them in the clouds. the data is about sport events from different categories and the user should be able to scroll into the time… this all should be filterable by categories like sport, region and so on… there are a lot of events and the most of the information we are presenting in this list is static.
I’ll think about the static json db. But I don’t think that the loads will be that high…for first
What I though about is to load data chunks based on the query, as you mentioned, with method call. And yes, methods feels to me, not like the way to do it. But with subscriptions I also not that happy, because of the reasons I already said, there should be a better approach to handle this data pulling stuff…
Hmm, what’s about the UI Issue? Where do you see problems if I just use the normal minimongo? Everything should work as expected, or do you mean not to overload the DOM?
I can’t say exactly the problem right now. But destroying and creating DOM is costly. And destroying Blaze templates also costly. So, you need to do some work.
We are also into the react hype now Seems to be the right UI platform that deals with Meteor Realtime Data like you would expect it. And it gives you the ultimate control over the DOM. as a little benefit, performance. And not to forget: it integrates nicely with flow-router and -layout. So if that works, great )
If you have a large number of Blaze templates in a infinite scroll list, its going to be ok at first but if you want to change the entire list at any point, you’ll have to destroy the templates that you had.
That takes an crazy amount of time as we’ve experienced with Blaze. React is generally faster on creation/destruction of templates than Blaze when it comes to large numbers.
However when we tested React on an infinite list, it seemed like the expanding subscription was re-evaluating all the elements from top (calling template helpers to check for changes) and that was slowing things down.
We didn’t dig in deeply on this however and would be interested to hear your results.
Ok, guys. I’ve create a demo app with React which creates 10000 Items and we get them with a forked version of publish-cache into the Local Minimongo. Then I am just applying it to the tracked Meteor Data and the list gets rerendered on new Data.
So for efficiency it’s better to load bigger chunks, because of the recalculations to create the Infinite List Items. It will be definitely the same for subscription data.
That Polymer 0.5 core-list can be interesting too, as it reuse dom LIs and change only part of it based on data.
So it generate like 3x what will fit in viewport and do not create/destroy any new while scrolling, no matter how long the list is. https://www.polymer-project.org/0.5/docs/elements/core-list.html#core-list
Ok, guys just did another example with out using the collection on the client side, just simple array… Actually for a Infinite list of static Items, you don’t need more
What it does: Just calling a method on the server side, get back an array and then I’m first pushing the new items to this.state.items array, and then making a single call this.setState({items: this.state.items})
Yeah, absolutely. Polymer is really nice. Under the hood both (core-list & Infinite React) are using some mathematical algorithm to render only the needed Items. But for me the React Component Architecture feels more naturally. And I like it, to have everything in one file. Even the styles.
Seems to be the right UI platform that deals with Meteor Realtime Data like you would expect it. And it gives you the ultimate control over the DOM. as a little benefit, performance. And not to forget: it integrates nicely with flow-router and -layout. So if that works, great )