Compute more on server-end and send less or inversely?

help
1

There is a large scale of records in MongoDB. On the client, I will display all these records after a complex (group,map,each,max, …) computing.
Now I have 2 choices to impletement it. The first one is to compute the results on server-end and send less data to client. And the second is, inversely, to send all these records to client and compute then.
Which is better?

2

Send the specific records required by each client to the client and use something like collections-helper to do the transforms client side.

If these complex computing is a once off transformation that isn’t affected by which client is looking or time, you could store the computed results in mongo as per the principles of denormalised document databases and then you don’t have to do any calculations at all.

3

Thanks, @mordrax .
It’s sound like a good idea. Before I posted this issue, an idea of saving results in Mongo came to my mind. I will try it.
But, as you know, Meteor is reactive. If I store the calculated results in Mongo, reactivity won’t work. Is there any way to keep the feature working well?:smiley:

4

There’s no silver bullet for reactively re-running complex computations. No matter how you slice it, running them often will put a bigger load on your app. If it’s feasible to send all of the relevant data to the client, that might be the easiest option, but if that’s not possible you can use Methods to load the data and refresh it manually.

Does this data actually need to be reactive? how fast does it need to update?

5

Thanks, @sashko
Here is the relevant data models.

  1. Model Practitioners with a field type.
  2. Model Licenses, which is embedded in Practitioners as a field whose type is Array and name is licenses, owns fields state and id which is unique. (Practitioners - Licenses is 1 - N )
  3. Model Jobs related to Model Licenses, has a foreign key named licenseId. (Licenses - Jobs is 1 - N).
  4. Model Workers has related fields practitioner_type and license_state.

Now I have a template Template_A which will display all jobs that are classified according to state (licenseId->license->state) and type (licenseId->practitioner->type).
So I enumerate all the 90+ workers and then find all the jobs related.
After this, I need group the related jobs by licenseId, and find out the last one by some order for each group.

//subscribe.
Meteor.subscribe('workersAndJobs');

//process.
_.each(workers,function(w,i){
   var jobs=findRelatedJobsSomeWay(w.practitioner_type,w.license_state);
   var gJobs=_.groupBy(jobs,'licenseId');
   _.each(gJobs,function(g,k){
      //find the last one.
      //do more things.
   });
   //do something else
})

On server, I publish all the workers and all the jobs owned by the current user.

Meteor.publish('workersAndJobs',function(){
   var licenseIds=[];
   Practitioners.find({userId:this.userId},{fields:{licenseId:1}}).forEach(function(p){
      _.each(p.licenses,function(lic,i){
         licenseIds.push(lic.id);
      });
   });
   return [Workers.find({}),Jobs.find({licenseId:{$in:licenseIds}})];
});

I think it’s very complex. I feel the Models should be redesigned. But I, at present, have no idea.
The Jobs contains several thousands records and subscription will take a very long time, and then compute for 10+sec.
This is the detail. Do you have some new way to it?
Thanks again.

6

Reactivity is not very need actually. It’s good to update it once a day.:expressionless:

7

Yeah if you only need to update once a day I’d use a cron job, with something like: https://atmospherejs.com/percolate/synced-cron