SpookyJS and Meteor

help
1

Hi,
after a lot of hard work I finally finished my first SpookyJS script to scrape some data from a website.
Now I want to use the result in a query against the database and/or call another Meteor.method with the result as an argument, but here I run into a problem that I couldn’t fix by myself for the last few hours, so any help would be appreciated.

Here is my spooky.on code:

spooky.on('fun', function (courses) {
  console.log(courses);
  Meteor.call('edxResult', courses);  // edxResult is never called... or at least it never gets to edxResult
  console.log('debug test');          // this line is never executed
});

The console output from console.log(courses) is definitely right:

[ 'course-v1:MITx+6.00.2x_3+1T2015',
'HarvardX/CS50x3/2015',
'course-v1:LinuxFoundationX+LFS101x.2+1T2015',
'MITx/6.00.1x_5/1T2015' ]

What I tried so far, calling the Meteor.method in which spooky runs with a callback, but that callback is executed immediately, not after spooky is finished (the whole spooky script takes around 10 secs)
I am not sure if using fibers/futures would help, I have never used them, so I would probably need a lot of help how to implement them for this.

2

Here is my solution that does, what I want, but its kinda hackish…

var edxCourselist; // global storage for the Courselist

edxResult: function() {
console.log('edxResult started');
Meteor.call('edxCrawling');
console.log('edxCrawling called, timeout next');

Meteor.setTimeout(function(){
  for (var i = 0; i < edxCourselist.length; i++) {
    var temp = 'stuff' + edxCourselist[i];
    Courses.find( { sessions: temp } ).forEach( function (item){
        console.log('got', item);
    });
  }
}, 20000);

edxCrawling: function () {
...
spooky.on('fun', function (courses) {
  console.log(courses);
  edxCourselist = courses;
  console.log(edxCourseList);
});

So basically I abuse a global variable and set a timeout, that is longer than the spookyjs script needs, which works, but is there a nicer way, with using callbacks? Because this looks like pretty bad code to me, or is it ok coding style wise?

3

this is more of a stackoverflow type question?