Http.get return corrupted json content

help
1

I am trying to use HTTP.get to make a remote api call then try to parse the returned json. However, the returned json object always has a corrupted content. Is it because the api call is https? Can someone please help?

I have put the snippet on meteorpad to demonstrate this issue.

http://meteorpad.com/pad/8LnKNnzusSNhzbQ5s/Leaderboard

Basically just pick a player then hit the “Add 5 points” link. The console output of result.content is corrupted somehow. If I tried to do JSON.parse(result.content), always get an unexpected token exception. Any help would be appreciated.

Thanks,
Bing

2

Maybe I am a friend of Cpt. Obvious here, but have you checked that the JSON is valid? You can use http://jsonlint.com for example.

3

yes, the json returned if I called the same url via browser and dump to http://jsonformatter.curiousconcept.com/

The thing is the data returned somehow is a bunch of corrupted characters somehow, maybe because the call is https and the returned json in content is somehow still encrypted?

It shall be very easy to reproduce via meteorpad.

any idea?
Thanks,
Bing

4

check out the returned content-encoding

I20150514-00:25:20.651(0)?   headers:                                                                                                         
I20150514-00:25:20.652(0)?    { server: 'nginx/1.6.2',                                                                                        
I20150514-00:25:20.654(0)?      date: 'Thu, 14 May 2015 00:25:20 GMT',                                                                        
I20150514-00:25:20.654(0)?      'content-type': 'application/json; charset=UTF-8',                                                            
I20150514-00:25:20.655(0)?      'content-length': '139',                                                                                      
I20150514-00:25:20.655(0)?      connection: 'keep-alive',                                                                                     
I20150514-00:25:20.656(0)?      'content-encoding': 'gzip',                                                                                   
I20150514-00:25:20.657(0)?      'last-modified': 'Wed, 13 May 2015 15:25:15 GMT' },                                                           
I20150514-00:25:20.657(0)?   data: null }
5

yeap, I just found that too. The response is gzipped, any library to unzip it?

Thanks,
Bing

6

Or how can I instruct http call to automatically gunzip the response if header has encoding as gzip? Thanks,

7

I’ve checked and confirmed that both https and http return the same data, so it is not an encryption issue.

Regarding gzip, I just checked the docs for the request npm module (which is what the http package uses behind the scenes) and it is supposed to the deflating for you so you don’t actually need to be doing that.

There is also the gzip: false that you can pass along with the npm request parameters option object, and I’ve tried that, but that still gets back a gzipped response so there is something fishy going on here.

You may have hit a bug!

8

Yes, it’s not an encryption issue but gzip issue. I agree that in theory meteor should have returned the gunzipped response since the response header indicate it’s a gzipped content, however, it didn’t work that way. Lucky me. I even tried the gb96:zlib library but have no idea how to make it work. :frowning: Also tried arunoda’s npm package to load zlib directly, but still no idea how to use zlib library. Any help?

Thanks,

9

Perhaps http://stackoverflow.com/questions/8880741/node-js-easy-http-requests-with-gzip-deflate-compression or http://stackoverflow.com/questions/10207762/how-to-use-request-or-http-module-to-read-gzip-page-into-a-string may help you.

But this is still very confusing, because according to this https://github.com/request/request

gzip - If true, add an Accept-Encoding header to request compressed content encodings from the server (if not already present) and decode supported content encodings in the response. Note: Automatic decoding of the response content is performed on the body data returned through request (both through the request stream and passed to the callback function) but is not performed on the response stream (available from the response event) which is the unmodified http.IncomingMessage object which may contain compressed data. See example below.

So basically, the response you see should already have been decoded!

10

Hey guys,

Sorry for reviving this, but I stumbled across it when I ran into the same problem and found a solution!

Http.get in it’s options can take npmRequestOptions if you pass an object here with gzip : true then it should decompress your results for you. Worked like a charm for me and I hope it helps you too!

11

Dude, thanks for saving me! I tried like 4 different packages and none of them worked. This method is even the most elegant!

12

I use this tool: JSON Formatter