Infinite HTTP requests server side question

help
1

Hi, i am building a website that streams live music and i want to implement something like this to display the current listeners for each show.

So I have a collection of radio shows with a field named currentListenerCount.

Just $incr on the count wouldnt work because people can easily fake listeners by opening on a bunch of tabs. I have a radio server running for the streams, so what i did is this:

-The streams are streamed from radio servers and you can get the current number of listeners by making a http request to the server, and it returns the number of listeners for a stream.

I want to keep making that http request and update the currentListenerCount with the http response from the radio severs, and do this for ALL document that is marked as a ongoing stream.

-keep the process running forever

How would i achieve something like this? Is there a better way to achieve the current listener count than what i have described?

2

In your server side code you can do something like this:

function getRadioStationStatistics(){
    // Do your HTML request here and update your DB accordingly
}
// Run the code that fetches the statistics once per hour or whatever you want
Meteor.setInterval(getRadioStationStatistics, 1000 * 60 * 60);
3

Thank you very much, that makes senseā€¦

Follow up question: How costly would it be to run that function every 30 sec , if i have something like ~500 shows playing.

And if it is too costly, would use like Redis for the listener stat make more sense?

4

I canā€™t really tell without running some tests, and I donā€™t have much time nowā€¦

I am certain that the HTTP request is expensive, but you can make it run asynchronously with something like this:

// By passing an empty callback we force the call to be run asynchronously
// You can still handle your callback result in there of course
Meteor.http.get(url, function () {});

Also I donā€™t know if doing this too often would block the server. When I had to deal with a similar issue, I created a server side route with iron router that itself did the request to the external service. That way I am sure each request is done on itā€™s own fiber without blocking the server, but I donā€™t know if thatā€™s a feasible solution for your case.

So a way to avoid blocking would look something like this

function getRadioStationStatistics(){
     Meteor.http.get(Meteor.absoluteUrl('/update_listener_counts'), function () {});
}
// Run the code that fetches the statistics once per hour or whatever you want
Meteor.setInterval(getRadioStationStatistics, 1000 * 60 * 60);

and in your routes

Router.route('/update_listener_counts', {
    action:function(req, res){
        // Do the real request and save to DB here
        // This is running in it's own fiber so it should be safe
        res.end();
    },
    where: 'server'
});

Again Iā€™m saying that I havenā€™t tested this in a heavy load production website so keep that in mind :smile:

5

Thank you for your input, using iron router is an interesting solution.

What about running the update function on the client?, so the listener stat is not stored in the database at all, it is directed send to the client template.

6

You mean with ajax? If they have CORS enabled and you donā€™t care about sharing said data among other listeners this feels the best solution

7

For information, Meteor.http calls are deprecated. The current syntax is HTTP.

Using an empty function() parameter just means no action will be taken when the HTTP call completes - it continues to run asynchronously and executes that (empty) function when it finishes.

Your options (server side) are:

(1). Synchronous HTTP call:
You can do this on the server by omitting the callback function entirely:

try {
  var result = HTTP.get('http://somewhere.com/endpoint/');
  // this line will be executed when the get completes (and "result" is available)
} catch (error) {
  // We got an error
}

I suspect that this is not what you want - 500 synchronous requests will take a l-o-n-g time to finish. Note that you cannot make an HTTP call like this on the client successfully, anyway.

(2). Asynchronous HTTP call:
You can do this on the client or the server by passing a callback:

HTTP.get('http://somewhere.com/endpoint/', function(error, result) {
  if (error) {
      // some error occurred
  } else {
    // "result" is available here
  }
});

From the documentation:

On the server, this function can be run either synchronously or asynchronously. If the callback is omitted, it runs synchronously and the results are returned once the request completes successfully. If the request was not successful, an error is thrown. This is useful when making server-to-server HTTP API calls from within Meteor methods, as the method can succeed or fail based on the results of the synchronous HTTP call. In this case, consider using this.unblock() to allow other methods on the same connection to run in the mean time. On the client, this function must be used asynchronously by passing a callback.

The advantage of doing this on the server is that you make one request to each radio station every 30 seconds. If you do it on the client and you have 500 clients connected, you will be making 500 requests to each radio station every 30 seconds. This sort of use will almost certainly violate their terms of service (you will probably want to check those regardless).

(You can get quite clever with async calls by running them in parallel, in which case the time to completion is the single longest time to complete. Use fibers on the server and promises on the client.)

My view is that if this is the only way to get that information, a server-side timer as originally proposed by @loupax is probably the best way to do it. The whole issue of reactively consuming REST endpoints seems to be cropping up a lot right now and Iā€™m not sure what the best approach is.

EDIT: Wrapped synchronous call in a try/catch block. Made discourse use my block numbering (grrr).

1 Like
8

Also Iā€™d like to add that I probably found a way to avoid creating custom routes

If I understood correctly, a way to ā€œtransform a syncronous call to asyncronousā€ is something like this:

var Future = Npm.require(ā€˜fibers/futureā€™);

Future.task(function() {
    // Make your HTTP request syncronously and add to DB
}).detach();

Will test it once Iā€™m off work, but if this works it really simplifies things

9

From fibers/future documentation on .detach():

an exception will be thrown to the event loop and node will probably fall down

Another option to consider is using .future() on your function. There is a good example of this for parallel execution in the Meteor code.

1 Like
10

And another good example here (note it also uses the deprecated Meteor.http syntax).

1 Like
11

I have it working now, thank you guys very much!