How to load huge amount of data for chart purposes

help
1

I have tried to use google charts & highcharts for my requirements. I am subscribing to the data before I start drawing the charts as shown below.

Meteor.subscribe('chartdata', {
    onReady: function (){
       //Once I subscribe the date from the server I'm drawing the charts
    }
});

I’m only loading 15,000 data (documents) from the mongodb as shown below. But for loading 15,000 points itself the application takes 5-10 seconds.

Meteor.publish('chartdata', function() {
    return Data.find({},{sort: {_id:-1}, limit: 15000});
});

The data subscribed is just one week of data. So then how to load millions of data in year’s time without consuming more time to draw the charts?

Apart from that I have below dobuts as well

  1. Using sql database high-charts are plotting 2 million data in no time in here - http://www.highcharts.com/stock/demo/lazy-loading
    Then why does meteor & mongodb taking more time to load 15,000 data itself? Am I doing anything wrong here (or) it’s the nature of meteor & mongo?
  2. Is it possible to load small small chunks of data while drawing the charts? If possible, can you point me to an example.

Thanks in advance

2

The sample on highcharts does neither load nor display 2 million data points:

https://www.highcharts.com/samples/data/from-sql.php?callback=jQuery31105937996230806712_1481046996692&_=1481046996693

It agregates them on the server and the client loads and renders the aggregated result.

You need to do something similar. Maybe aggregate the data every day/week/month/year and put the aggregated data in new collections.

1 Like
3

@macrozone

Sorry just a wrong link. http://www.highcharts.com/stock/demo/data-grouping. In here you can see, they are getting 52,000 database values in no time.

From your advice, I can try to aggregate the data points on the server before the client loads it. But I would like to know why meteor & mongodb taking more time to load even less data points (say 15,000)

4

It’s not database backed. It’s reading from a single json file found at https://www.highcharts.com/samples/data/large-dataset.json

5

@macrozone @kimberger

So, basically you are saying getting huge data (say million records) from any database is time consuming? Then what’s the better way to load small chunks & chunks of data from mongodb?