Best practice for drilling into more detail on a single record and pulling more fields from the database?


#1

I am displaying a limited amount of fields in a table grid and the user can click to go to a new route that will show the full record. When pub/subbing the data for the grid, I pull up just a few fields, but the detail view should re fetch the document from mongo and pull up more fields I didn’t previously have.

Is there an example or writeup on a best practice for this?

I’m currently doing a findOne in the route and passing the document to the template, but I am unable to get fields that weren’t already gotten from my subscription for the table grid.


Subscriptions: One giant Monolithic or multiple small? which is preferred?
#2

create a sub template for the detail records and subscribe to the detail data from that template.


#3

I was looking for similar thing.

If sub-template is subscribing each time the ‘selected item’ changes in the parent template - can Meteor/Mongo sustain that kind of activity?

The reason is: If there is a list or table of entries in the parent item, and if users keep clicking on each item one after the other (perhaps to explore or find something they want) - each time changing the subscription in the child template sounds like heavy-duty.

May be I am mistaken, but are the subscriptions a light-weight concept in Meteor or are they still heavy-duty ones (like the old database connections, where we recommend to pool the connections and avoid frequent connection-close cycles)?


#4

Although it all depends on specific use-cases, it should be fairly safe to assume this is a better way than preloading the data.

Yes it can change frequently, but subscription and data request is very intelligent within meteor. And if you plan them right, subscribe to data only when necessary, and subscribe to the set (specific docs and specific fields in those docs) of that data that you exactly need, you’ll be good.

After all, only the collection data (with very minor overhead) is passed on the wire and it is the least amount of data that you can ask from a server in a web app.

Connection pooling is another concept that is between the server and the database. Mongo world is a lot different than traditional database connections that you may be used to from the mysql world you probably are referring to.

Opening connections to the database and reusing it wisely is meteor’s internal job. Meteor also employs powerful mechanisms to keep both the server and client side collections in a good state for the app with minima change and maximum efficiency.

So by all means, do go for a parent/detail subscription pattern and subscribe to the detail from your template instance, not your router.

Also, don’t hesitate to post in some code here for further assistance.


#5

So, I have one “page” of the app /showRecords that will show, say 100 records in a table, and each will have a button to “view full record”, then I want that click to go to a completely new route /showSingleRecord/3494.

is this where i would need parent child templates? or just2 templates?


#6

I created a working demo to show you how.


#7

This way, you do not need any hacks. Your new route /showSingleRecord/3494 should subscribe to have only one detailed record (seems _id=3494). You can do this simly with IronRouter.


#8

It works fine, but what if I want to show ‘extended’ inline details? I should take additional fields for the currentRestaurant from server! So I suppose we cant separate templates for list and details, but also can subscribe for details info like:

Template.Restaurants.onCreated(function() {

  this.state = new ReactiveDict;
  this.state.set('currentRestaurant', null);
  
  this.detailedSub = null;
  
  self = this;
  
  detailsSubHandle = _.debounce(function(currentRestaurant) {
    if(self.detailedSub) {
      self.detailedSub.stop();
    }
    self.detailedSub = Meteor.subscribe('restaurantDetails', currentRestaurant);
  }, 1000);
  
  this.autorun(function() {
    currentRestaurant = self.state.get('currentRestaurant');
    detailsSubHandle(currentRestaurant);
  })
});
// do not forget to destroy this.detailedSub when onDestroyed calls

In this sample we call detailsSubHandle every time user pick an item, but subscription fires only after 1000ms (as a sample, see underscore debounce) - so we dont overflow server calls and ui looks solid. Using this method we created complicated realtime filters for a user interface with a good productivity.


#9

in my demo, the selected record _id is passed to the details inclusion template. So you can simply query and pull whatever fields you want. Setup two subscriptions, one for the “list” (restaurant in my demo) and one for the “details” (menu items in my demo).


#10

But what if user started to click on a each list’s item? Everytime he does it we have to render new template and this template runs subscribtion? It is very expensive way! I tried. I can click items with just 500ms delay and after 5-6 clicks you’ll see the hell. Server will pull a new recordset for detailed document with this delay to. Moreover the client UI becames really slow.


#11

It’s a tradeoff. You have two options as far as I can see:

  1. in your original publication for the doc, pull more fields. Then as they drill down, you already have the data in their local collection and only need to show it.
    Advantage: faster drilldown.
    Potential Disadvantage: slower initial load if you have a lot of detail data to load.

  2. do what I suggested: load the details dynamically as they drill down.
    Advantage: faster initial load of the list
    Disadvantage: slower drill-down.

or you can combine the two approaches in various ways i.e. during the initial load, get the fields for the first two levels. Top level data and one drill down level. Then if they drill down one level it will be snappy. If they need to drill down again, you’ll have to load data again, but you could load several levels again i.e. all the fields need for level 3, 4, 5.

It really depends on how much data you have to show. If it’s just a few fields, include them in the initial sub so you’ll have the values ‘pre-cached’ for the drill-down. But it’s it’s a massive amount of data, then you’ll have to think about where you want to pay the performance cost. It’s not really a Meteor problem; you’ll have this same compromise to make with Rails or any other distributed application framework.

It’s just like loading a city/state list for an address form. If you wait until they pick country, you have to load the cities and there is a small delay. Or you can pre-fetch all the city/state data for every country, but maybe that slows page load esp. on mobile.

I’m surprised the drill down time is 500ms. We’ve seem much better performance. Our apps are near real-time in syncing data from clientA to server then down to clientB. Very minimal delays.

Where are you hosting? how much data is in the collection (document count), how much data are you pulling (a few fields? or 30kilobytes? more?, are your query fields indexed in mongo?


#12

We are on DO with 4GB mem ($ 40 / mo pack). ComposeIO with oplog for Mongo. It works fine, but we had an issue with fetching the data (100-150 document in a subscribtion and 5-8 fields in each document). Users could interact with 8 different filters, so our DB queries may be really differents. As I pointed above, users just click filters options as they wants. So the filters are simply reactive dict, and while user iterates filters we just block main list view and call subscribe handler. Subscribe handler is wrapped into _.debounce(500) and that is mean client could call a server only if no filters are changed during 500ms.
Working without this hack messes ui and asks server to much time, even if you optimize server-side publish query as possible as you can.
I agree that it depends on the case. But dont you agree that fetching a list of all the cities after user have typed only first country letter arent so optimzed?


#13

I may not have clearly explained what I am experiencing… I have simplified my app to isolate this particular issue, and created a meteorpad for it:

http://meteorpad.com/pad/xKnuw38ce64xCcqbR/DemoDrillDown

The detailed explanation is shown on every page, so open the app in a separate browser and scroll down to see the results of each route/page. I hope this is clear enough, any feedback will be greatly appreciated!


#14

Issue solved, sort of. at least I now know why I was not getting my new fields when trying to drill into the detail, it’s the “known issue” with DDP, which some are aware of, but many wish the docs would address it more clearly. The fields I was trying to get in the detail route were sub fields for which my initial subscription already pulled some sub fields from. And, once you pull some fields from a sub field, you are then unable to add other fields on subsequent subscriptions. https://github.com/meteor/meteor/issues/998 https://github.com/meteor/meteor/issues/903 http://stackoverflow.com/questions/22543148/meteor-users-not-synchronize-published-sub-fields-of-profile


#15

I put together a solution here:
http://meteorpad.com/pad/tC6bmtBpjwGWsnw4q/DemoDrillDownSolution
It uses the concept described in suggestion #5 of this comment:


#16

I starting working on your project but thought it was a bit confusing how you have to refresh to change the data. Also, instead of having all those if isClient and if isServer blocks, you could just move the code into client and server folders. Maybe that is just a limitation of Meteor Pad, anyway here is how I reorganized your code:

Also minor note but Meteor.Collection was renamed Mongo.Collection in 0.9.1. The change is currently backward compatible, however you should switch to using Mongo.Collection for any new projects.

I thought your demo was a bit messy, again, maybe because of MeteorPad, so I added similar publications and subscriptions into this drill-down demo. Source code is pushed to a public git repo. Click reinit data to start. You can see how the details are dynamically augmented into the collection as you drill-down. You could easily extend by adding your router code. Just put the subscriptions in router onWait hook

Router.route('/chart/:_id', {
  waitOn: function() {
    return Meteor.subscribe('chart', id);
  },

Let me know if you have any questions.

http://drilldowndemo.meteor.com/


#17

Thanks for taking the time to look at my example. While I was originally looking for the best practices, it turned into a discovery of a DDP limitation (which is definitely known, but not too well explained in the documents)… so now I am basically looking for good ways around this limitation, and also hoping they might change this some day :slight_smile: … so, I took your example and forked it and added sub elements so that it will illustrate this DDP issue. Basically, what I’ve learned is that subscribing to additional fields that were not pulled up to the client initially works fine (as your example shows) but only works when you are not working with nested documents, like:

{name: “Italian”, phone: “555-1111”, status: “*****”, address:{street:“3 First St”,city:“Denver”, state:“CO”}}

If your documents are structured like this, and you wish to first publish just the address.city, and then later subscribe to the address.street and address.state when the user drills down for more detail, you are unable to get the additional fields from address, because meteor/DDP already thinks you’ve fully subscribed to the address element, even though you have only pulled a subset of fields from the address.

my changes are here:

and, if you look specifically at the file that publishes the restaurant and restaurants publications, you’ll see my comments where you can see this happening.


#18

OH really? wow that’s interesting. I’m going to research that a bit myself. We have a lot of nested docs in my application. Thanks for explaining it. I understand the issue now. I’ll check out the fork!


#19

so if I understand maybe this is a better demo?

Meteor.publish("restaurant", function(id) {
  return Restaurants.find({_id: id}, {fields: {_id: 1, name: 1, phone: 1, status: 1, "address.city": 1}});
});

Meteor.publish("restaurants", function() {
  return Restaurants.find({}, {sort: {"status": 1}, fields: {_id: 1, name: 1,"address.state":1}});
});

We get ONE address field for the top level list (address.state) which prevents us meteor from loading the rest of the address fields during the drill-down?


#20

You are right, after Client has been subscibed to any publication from you sample, next suscribtion won’t pull address field again. So Client-side address field will be the same.