Using PhantomJS to process http requests

I need to process a response from a request that HTTP module cannot (the response includes JS script).

I’m trying to use PhantomJS but I see conflicting instructions on how to install it and how to run it.
Basically, I did this in my app folder:

meteor npm install --save phantomjs

but now I don’t know how to proceed. All I want to do is use PhantomJS with a URL (with GET) that I know works in the browser but do not for HTTP (as the redirect returns with JS script).

Anyone?
Thanks
Thanks

bumping again. Anyone?

So, the scenario is that you are sending a HTTP request, like a GET to an external server. The response to this HTTP request is a message whose body includes JS code? Like a 200 OK is coming back, with a result of object type, and within this object there are fragments of JS?

Are you sure it is not an XML/HTML object structure being returned and your problem is you need to parse this?

Sorry - don’t really get your use case.

Tat

Took me a little bit time to find it but here is the flow:

I use a GET url w/ parameters
server returns with 200 and the response is following html, where is the next one executed and is some internal URL:

How do I continue from here?

<!doctype html>
<html>
    <head><script type="text/javascript" src="<JS-SCRIPT>"> </script>
        <meta charset="utf-8">
    </head>
    <body>
     <script>
       var xhttp = new XMLHttpRequest();
       xhttp.addEventListener("load", function(a,b,c){
         window.location.reload()
       });
       xhttp.open('GET', <URL>, true);
       xhttp.send();
     </script>
   </body>
  </html>

hey,

going forward, you probably want to hit reply on the person’s post. otherwise no notification so impossible to know if someone has comeback to you or not.

To be honest I still don’t understand your use case. This is how I think of HTTP requests. Below i’m going to use an example of doing some stuff with a file on github for example.

(1) Client side - some interaction which triggers action that the user wants to do. this action is the user asking the app to do something which includes the get request as the user needs info from somewhere outside our app. So on client side, i have a bit of JS which looks something like so:

// user clicks a button or something...
handleTouchTap() {
    // ... stuff
    // ... now we need some stuff from github...

    getAccessToken()
      .then((result) => {
        updateFilePromise(
          result.data.token, repoName, filePath,
          message, content, name, email, sha,
        )
          .then((result) => {
            if (result.headers.status === '200 OK') {
              // do some stuff after getting back info
            } else {
              throw new Meteor.Error(result);
            }
          });
      })
      .catch((err) => {
        throw new Meteor.Error(err);
      });
  }

(1b) Still on client side, i have a helper of some variety, like below. You don’t have to do this, i’m just usign it as a wrapper for a promise.

export function getAccessToken() {
  return new Promise((resolve, reject) => {
    authorizationGH.call({ ghAction: 'getAuthToken' }, (err, result) => {
      if(err) {
        reject(err);
      } else {
        resolve(result);
      }
    });
  });
}

(2) this goes to server side on out app. here, we have a method like so. Obviously below is a post request, but the below method gives me back an auth token. Then i use another one of these to get whatever info I need.

export const authorizationGH = new ValidatedMethod({
  name: 'gh.authorization',
  validate: gh.omit(['ghInformation']).validator(),
  async run ({ ghAction }) {
    if (Meteor.isServer) {
      switch (ghAction) {
        case ('getAuthToken'): {
          const jwt = require('jwt-simple');
          const cert = fs.readFileSync(Assets.absoluteFilePath(<paht-here>'));

          const now = Math.round(Date.now() / 1000);
          const payload = {
            iat: now,
            exp: now + 600,
            iss: Meteor.settings.private.github.intId,
          };
          const token = jwt.encode(payload, cert, 'RS256');

          // auth
          const method = 'POST';
          const url = 'https://api.github.com/installations/'
            + Meteor.settings.private.github.instId
            + '/access_tokens';

          const headers = {
            'User-Agent': 'Meteor/1.0',
            'Accept': 'application/vnd.github.machine-man-preview+json',
            'Authorization': `Bearer ${token}`,
          };

          return await HTTP.call(method, url, { headers });
        }
        default: {
          throw new Meteor.Error(
            'gh.authorization.failure',
            `Method ${ghAction} was not recognized...`,
          );
        }
      }
    }
  },
});

Basically on the return from the HTTP request, you would then parse the HTML fragment (or whatever) the server sent you back.

Specific to dealing with HTML, on the result, i would do something like this on the response. You may need a more sophisticated parser if you are getting funky HTML/JS back.

.then((result) => {
  let parser = new DOMParser();
  const xml = parser.parseFromString(result.content.trim(), 'text/xml');
  const foo = xml.getElementsByTagName('string')[0].innerHTML;

It is probably also worth using something like DOMPurify to not get bitten in the backside.

The above works for my use case. I would totally spend some time asking around some more if you are not sure of your requirements. Also look into using rate limiters.

Most of the stuff above is from the meteor guide. You should look it up.

Thanks.

Tat

You need to use a fragment URL. This tells a bot/browser in your case phantom to wait for the JS to render before pulling back HTML. Or something like that.

http://localhost:3000/?escaped_fragment=

Try that :wink:

If you look in your element debugger in Chrome, you’ll now see all your page HTML as you’d expect.

Thanks. I’ll try the two suggestions and report back

So I got this working. You need to install the Node PhantomJS package, which is completely separte from Meteor.

You run that on Localhost:3000.

Then change Meteor to Localhost:4000.

Then run Phantom and parse Meteor. Like this: http://localhost:3000/http://localhost:4000 which, will return your pre-rendered content.

As you can see here on my FB page, I’m FINALLY pulling SEO meta / og data tags!

https://www.facebook.com/skyroomsio/

I am using these two packages:

dferber:prerender
manuelschoebel:ms-seo

Finally it’s frigging working.