Client Method async call to Server async HTTP call returns undefined?


#21

Yeah, you are right. One more “thanks”. :grinning:


#22

OK, I’ve updated the repo. Sadly, it’s not as easy to use as it was, and the frequency endpoint has been removed completely. However, what works, works :wink:


#23

Thanks, @robfallows .

I am not sure what I am doing wrong but when I run APIKEY=“my-api-key” meteor --settings settings.json, console says ’APIKEY’ is not recognized as an internal or external command, operable program or batch file.

Do I need to do this?

Please suggest.


#24

OK. You’re using Windows. On PowerShell you will need to do

$env:APIKEY="your-api-key"
meteor --settings settings.json

In cmd

set APIKEY="your-api-key"
meteor --settings settings.json

#25

Thanks, @robfallows.

I got this running by setting process.env.APIKEY in ukgrid.js on server. Cheers!


#26

Cool - just remember not to publish that code to a public repo, or your API key will be exposed!


#27

Yeah… I understand that. Thanks.


#28

Rob, I added a new topic two days ago; no responses yet.

Can you please help on that as well?


#29

Hi, Rob.

I need to query a collection and store all document objects into an array. I’ve been looking around for a solution on this without success. Is there any builtin way in Meteor for this?


#30

An array in Javascript?

const myArray = MyCollection.find(...).fetch();

Where ... will be your query if you are selecting a subset of documents.


#31

Thanks, Rob! I got around that a minute after I posted my question. Your answer confirmed that I found the right thing.


#32

Hi, @robfallows.

I am back with new problem. :laughing:

Actually I am looking for suggestion how to achieve this. Here is what I need to do:

  1. Client sends a request to server - a list of URLs along with some other info to create static webpages (htmls) on a given directory.
  2. Server accepts the requests and starts creating html files for all the received URLs one by one.
  3. While request is being processed on server, keep showing progress status on client real time (to all the logged in users on the app) - How many URLs have been converted to static pages out of the total URLs sent to server.

I am not sure how I will achieve #3 here. I know Meteor instantly updates client w/o page reload for changes but I am not able to figure out how a queue processing on server will update client. Please suggest what should be the way to do this?


#33

So I googled around it and learnt that I should use Rocketchat:Streamer package.

I am trying to use this package but not able to get it working per my needs. As long as I keep streamer initialization and keep using streamer.emit() and streamer.on() from inside lib folder of my project, I can send and receive messages via browser console. As soon as I try to use .emit() from server folder and .on() in client folder, it just stops working. I need to send progress data from server to all logged in clients.

I may be doing some silly mistake to get that package work from server to client but I am not sure what that may be.

Please help!


#34

Finally, I got it working. I am yet not sure what I was doing wrong earlier. I moved client and server code from lib file (leaving streamer initialization still in lib folder) in respective folders and it started working.

Thanks. Cheers!


#35

Glad you found a solution. You could have used Meteor’s pub/sub to do this as well.


#36

Thanks, Rob.

pub/sub means publish at server and subscribe at client. Right?

By what I’ve read about it so far, that works when you are working with collections. Server keeps inserting docs in collection and client keeps updating itself in real-time. Is there a way using pub/sub w/o using collections?


#37

There is also one more thing that I need suggestions/help on.

I call the server method supplying an array. Server method needs to

  1. read each array element
  2. make a http.request call to read URL from array element
  3. retrieve URL’s page content via http.request()
  4. finally save URL in DB.

When server is done doing this on all array items, it needs to send a flag back client.

I know Meteor.Async() helps doing things synchronously. should I apply Meteor.Async() on http.request calls? Here is my snippet:

.
.
.
for(var i = 0; i < selectedPages.length; i++){
        var pageDetails = allPagesDetails.filter(obj => obj.pageUrl === selectedPages[i] );
        /*var pageDirPath = url.parse(pageDetails[0].pageUrl).pathname;
        var pagehost = url.parse(pageDetails[0].pageUrl).host;
        console.log(pageDirPath);*/
        // console.log(pageDetails);
        if(i == selectedPages.length -1) lastPage = pageDetails[0].pageUrl;

        getPageContent.getPageContent(pageDetails, function(err, response){
          if(err){
            console.log(err);
            return
          }
...
...
Fiber(function() { 
              try {
                PageSitemaps.insert(response.pageDetails[0]);
                sendMessage({
                  event: "processBuildProgress",
                  pageUrl: response.pageDetails[0].pageUrl,
                  statusCode: response.statusCode
                });
              } catch (error) {
                console.log("Error while Saving to DB: "+error);
              }
            }).run(); 
}
}


//================ getPageContent ===============
function getPageContent(pageDetails, callback){
    /*var pagehost = url.parse(pageDetails[0].pageUrl).host;
    var pageDirPath = url.parse(pageDetails[0].pageUrl).pathname;*/
    var options = {
        host: url.parse(pageDetails[0].pageUrl).host,
        path: url.parse(pageDetails[0].pageUrl).pathname
    }
    // console.log(options);
    var request = http.request(options, function (res) {
        var data = '';
        res.on('data', function (chunk) {
            data += chunk;
        });
        res.on('end', function () {
            // console.log(res.statusCode);
            // console.log(res.statusMessage);

            var pageRequestResponse = {
                statusCode: res.statusCode,
                statusMessage: res.statusMessage,
                content: data,
                host: options.host,
                pagePath: options.path,
                pageDetails: pageDetails
            }
            for(i in res){
                // console.log(i);
            }
            // callback(JSON.parse(data));
            callback(null, pageRequestResponse);
            // return data;
            // savePage(data);
        });
    });
    request.on('error', function (e) {
        console.log(e.message);
        callback(e.message, null);
    });
    request.end();
}


#38

Yes …

You can consider a subscription as the client telling the server to do something. The publication can do whatever you want. Most times that will be a query against a MongoDB collection - but it doesn’t need to be.

The other thing that’s expected of pub/sub is: “while you’re doing that thing, keep me informed about progress while you do it.” Again, you’ll usually just use the inbuilt livedata in Meteor to ensure a collection’s changes are available to the client - but it doesn’t need to be.

… and no :wink:

The way server-side changes are communicated to the client is via minimongo. However, it’s important to note that you don’t need an actual (disk-based) MongoDB collection for this. Instead the publication uses the pub/sub API to populate a client-side minimongo collection.

So, the publication could be written like this:

Meteor.publish('buildFiles', function(urls) {
  // skipping the sanity checks and error checking for brevity
  urls.forEach((url, index) => {
    const data = HTTP.get(url);
    buildHTML(data);
    this.added('dummyCollection', `seq${index}`, { index, url });
    this.ready();
  });
});

Note that the above code processes the urls sequentially. You could use callbacks or Promises to process them concurrently.

As each url is processed, a new document is added (this.added) to the client-only collection, dummyCollection, and made available to the client (this.ready). The document imitates a MongoDB document, so needs an _id ('seq0', 'seq1', …). Otherwise, my example just has an index field and the url. You can, of course, add whatever you want.

On the client, you need to define a matching collection and subscribe in the normal way. Note that you don’t define this collection anywhere on the server - it doesn’t exist there.

const dummy = new Mongo.Collection('dummyCollection');

Template.buildFiles.onCreated(function buildFilesOnCreated() {
  this.subscribe('buildFiles', ['url1.com', 'url2.com', 'url3.com']);
});

Template.buildFiles.helpers({
  buildProgress() {
    return dummy.find();
  }
});

You can also make use of autorun and have reactive parameters in the subscribe, exactly as with any other subscription, for example for resubscribing with another url list.


#39

Thanks a lot, @robfallows!

It took me sometime to understand your solution (novice here :smile:). I couldn’t re-write whole whole process as this but your snippet gave me insight to use HTTP.get() in-line so all pages are read sequentially. I did a small tweak and tested; it’s working so far.

I will update you once I am fully done. Thanks again.


#40

Hey, @robfallows.

I am facing another challenge here. As I wrote earlier, I am using HTTP.get() to read URLs and their content. HTTP.get() call is inside a loop which runs over an array of URLs. As long as array length remains moderate say not more than 150 or so, my code runs fine and server keeps sending progress data to client vai Rocketchat:Streamer package.

But when array length is higher say around 1000, then my code stops arbitrarily and server stops sending data to client. It neither throws any error on server/client; it just hangs. If I bypass HTTP.get() execution, code runs completely. Here is the code snippet:

for(var i = 0; i < selectedPages.length; i++){
        var pageDetails = allPagesDetails.filter(obj => obj.pageUrl === selectedPages[i] );
        var pageDirPath = url.parse(pageDetails[0].pageUrl).pathname;
        var pagehost = url.parse(pageDetails[0].pageUrl).host;

        if(i === (selectedPages.length - 1)){
          lastPage = true;
        }

        try {
          const data = HTTP.get(pageDetails[0].pageUrl);

          var statusCode = data.statusCode;
          var pageContent = data.content;

          /*var statusCode = '200';
          var pageContent = '<h1>Dummy Content</h1>';*/
        } catch (error) {
          throw new Meteor.Error('Oops...', 'Some error happened while reading '+pageDetails[0].pageUrl);
        }
        

        var dirs = pageDirPath.split("/").map(function (ele) {
          if(ele != ""){
            return "/"+ele;
          }
        }).filter(arr => arr);
        for(let j = 0; j < dirs.length; j++){
          if(j > 0) dirs[j] = dirs[j-1]+dirs[j];
        }
        console.log(dirs);
        dirs.map(ele => checkDirectorySync(targetDir + ele));
        var pageDir = targetDir + pageDirPath;
        console.log(`pageDir: ${pageDir}`);
        checkDirectorySync(pageDir);
        fs.writeFileSync(pageDir+"/index.html", pageContent);
        
        // ==========================================
        try {
          PageSitemaps.insert(pageDetails[0]);
          sendMessage({
            event: "processBuildProgress",
            pageUrl: pageDetails[0].pageUrl,
            statusCode: statusCode,
            buildInitiator: initiator[1],
            buildInitiatorName: initiator[0],
            selectedPagesNum: selectedPages.length,
            isLastPage: lastPage,
            currentPageNum: (i+1),
            buildInitTime: buildInitTime,
            pageProcessingTime: new Date(),
            targetSite: targetSite
          });
        } catch (error) {
          console.log("Error while Saving to DB: "+error);
        }
        // ==========================================
        if(lastPage){
            console.log("Build processing is DONE.");
            Builds.update(currentBuild, {$set: {processingStatus: 0}});
            return true;
          }
      }

This loop is part of a method on server which is supplied the URLs array from client.

Thanks in advance!