Client Method async call to Server async HTTP call returns undefined?

ryuzaki · May 10, 2016, 1:57pm

Hey

Just a quick nooby question. I’m playing about with meteor HTTP calls on the server and returning them to the client. Got it working, however when I added a callback to both the client Method call as well as the server HTTP call I’m getting undefined.

Is this just a bad idea to have both async? If I switch the server to sync it works fine.

Here’s the code snippets. I’m using react for the client frontend:

client:

Meteor.call('pollUrl', (error, result) => {
            console.log(result);
            console.log(error);
            return result.statusCode;
        });

server:

if (Meteor.isServer) {
    Meteor.methods({
        pollUrl: () => {
            HTTP.call('GET', 'http://api.giphy.com/v1/gifs/trending?api_key=dc6zaTOxFJmzC&rating=g&limit=12', {}, (error, result) => {
                console.log(result);
                return result;
            });
        },
    });
}

Cheers in advance!

robfallows · May 11, 2016, 10:50am

When you invoke your async version of the method it returns undefined almost immediately, because the code continues to be executed while the async operation is waiting. This is what your method actually appears to do from the perspective of the client’s call:

Meteor.methods({
  pollUrl: () => {
    return;
  },
});

That near instant return of undefined satisfies the call and the subsequent return of the async operation does nothing.

Just a final cautionary point: using the fat arrow notation for a method (or publication) loses the expected context of this for the method (or publication) body. If you want to make use of any of the current user data (for example) you will need to use the ES5 syntax:

Meteor.methods({
  pollUrl: function() {
    // your code here
  },
});

ani1219 · January 8, 2018, 9:47am

Hi, robfallows.

I am also running into similar problem. I am calling a server method from client. The server method function runs an HTTP.call() to read an external XML and then uses xml2js.parseString() on its callback function to convert XML data to JS. I want to return JS object obtained within xml2js.parseString() inside HTTP.call()'s callback function:

Meteor.methods({
	readSiteMapXml(){
		HTTP.call('GET',
            configJSON.SOURCE_SITEMAP_XML,
            {},
            function(callError,callResponse){
            	if(callError){
            		console.log('callError ',callError);
            	}else{
            		// console.log('callResponse.content.loc ',callResponse.content);
	                xml2js.parseString(callResponse.content, function (jsError, jsResult) {
	                	if(jsError){
	                    	console.error('errors',jsError);                		
	                	}else{
	                		const util = require('util');
// console.log('Result - ', util.inspect(jsResult.sitemapindex.sitemap[0].loc, false, null));
	                    	return jsResult.sitemapindex.sitemap;
	                    	
	                	}
	                });
            	}
            }
        );
	}
})

The client gets undefined instead. This is client:

Template.Builds.events({
	"click .getxmlinfo": function(){
		$(".flx-loader").css('display', 'flex');
		Meteor.call("readSiteMapXml", function(error, response){
			$(".flx-loader").css('display', 'none');
			if(error){
				console.log(`Error: ${error}`);
				return;
			}
			console.log(response); //returns undefined
		})
	}
})

Based upon my searches on this, I understand that server function immediately returns undefined to client and continues executing HTTP.call and then xml2js.parseString() on callbacks.

Please suggest me how can I get response of xml2js.parseString() in above snippet on client?

robfallows · January 8, 2018, 12:24pm

Can you please edit your post and put your code between triple-backticks, like this:

```
your
code
here
```

minhna · January 8, 2018, 2:43pm

I created an example of async method call at https://github.com/minhna/meteor-react-ssr.
It’s on the test page. You can clone and go to the /test page.
Sorry for my bad English.

robfallows · January 8, 2018, 3:13pm

So, you could do something like this on the server:

Meteor.methods({
  readSiteMapXml() {
    import xml2js from 'xml2js';
    xml2js.parseStringSync = Meteor.wrapAsync(xml2js.parseString, xml2js);
    try {
      const callResponse = HTTP.call('GET', configJSON.SOURCE_SITEMAP_XML, {});
      // console.log('callResponse.content.loc ',callResponse.content);
      jsResult = xml2js.parseStringSync(callResponse.content);
      const util = require('util');
      // console.log('Result - ', util.inspect(jsResult.sitemapindex.sitemap[0].loc, false, null));
      return jsResult.sitemapindex.sitemap;
    } catch (error) {
      throw new Meteor.Error('oops', 'Some error happened');
    }
  },
});

or, you could use one of the Promisified versions of xml2js, which would allow you to get rid of the Meteor.wrapAsync() and use an async method.

ani1219 · January 8, 2018, 4:44pm

Thank you so much, @robfallows!

I commented the first line in the function to import xml2js and I got response on client.

I am a novice and just wading my way through Meteor. If you could explain a bit how your solution is working, that will be greatly helpful and appreciated.

robfallows · January 9, 2018, 11:16am

So, Meteor removes the complexity of returning data from an asynchronous code block by allowing you to write what looks like synchronous code (what you typically use with Python or PHP).

Traditionally, it’s done this using Fibers, but we can also use Promises and async/await. The example I gave made use of Meteor.wrapAsync(), which may be used to “wrap” any asynchronous function with a callback signature of (error, result) in a Fiber and allow it to be used the same way.

Fiber-wrapped functions effectively cause the execution of the event loop to wait for the result to become available.

ani1219 · January 9, 2018, 11:52am

Thanks, @robfallows! Great explanation. I also tried to read on web about this.

As I told you that I am new on Meteor, I am stuck into another road block. Since I am new here too, I am yet not allowed to log new items in this forum.

I want to check and create new directory inside “Private” folder and then create some files on the fly. I am using fs (var fs = Npm.require(‘fs’) to do this but no matter what I try, no luck! Not even able to access private folder -

var files = fs.readdirSync('../private/');

It throws error -

Error: ENOENT: no such file or directory, scandir 'D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\private'
    at Object.fs.readdirSync (fs.js:904:18)
    at checkDirectorySync (server/methods.js:102:17)
    at methods.js (server/methods.js:115:1)
    at fileEvaluate (packages\modules-runtime.js:343:9)
    at require (packages\modules-runtime.js:238:16)
    at D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\server\app\app.js:461:1
    at infos.forEach.info (D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\server\boot.js:414:13)
    at Array.forEach (<anonymous>)
    at D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\server\boot.js:413:9
    at D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\server\boot.js:463:5
Exited with code: 1
Your application is crashing. Waiting for file change.

Any help will be greatly appreciated.

robfallows · January 9, 2018, 1:19pm

A couple of things here.

Firstly, the private folder is used for static assets and is created along with the rest of the collateral belonging to your app - you shouldn’t use it as an upload location. Check the Assets section of the docs.

You should avoid using any fs.xxxSync calls - they lock the nodejs event loop up, completely stopping any other nodesj code until the function completes. On a moderately busy system, that can make the server code almost unusable.

ani1219 · January 9, 2018, 1:53pm

I see… thanks for the insight!

I need to read an xml and create files using data from the xml in a
directory within my app. What are the other options I should go for?

robfallows · January 9, 2018, 2:22pm

Just to be clear, there’s nothing actually preventing you from using the private folder. It’s just that the contents are packaged as part of the build. That, in turn, means that the content may be nuked on a redeployment. In fact, depending on how you choose to host your production code, you may find your local file storage is entirely disposable. For that reason, S3 is often used for permanent file storage.

However, if you want to use local storage, you can use private or any server folder outside of the application root or application build phase.

I wonder if you got your error because you didn’t wrap your fs call in a try/catch and the directory didn’t exist?

ani1219 · January 9, 2018, 5:37pm

You should avoid using any fs.xxxSync calls - they lock the nodejs event loop up, completely stopping any other nodesj code until the function completes. On a moderately busy system, that can make the server code almost unusable.

The app does need to first create those files before it goes on to do anything next. Once all files are written, then only app needs to execute further. So async function here may not be helpful. Thoughts?

Assets section of the docs

Thanks for sharing the link. I also looked through the Assets api documentation. It also says there that “Note that assets are read-only.”. I reckon anything and everything inside Private folder is “assets”?

I think, using Private folder to write those files should be fine because those files will be created when app will be used by users. So when deploying on production, Private folder doesn’t need to have those files.

Despite all my extensive search, I am not able to find a way even to access Private folder from Server folder, let alone creating a folder or file there.

I am looking for few answers here:

what’s the right way to access project directories from server folder? I do have a “private” folder in my project that currently consists of a config.json. Doing this:

function checkDirectorySync(directory) {
  try {
    fs.statSync(directory);
  } catch(e) {
  	console.log(e);
    fs.mkdirSync(directory);
  }
}

checkDirectorySync("../private/xmls");

results into this error:

{ Error: ENOENT: no such file or directory, stat 'D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\private\xmls'
    at Object.fs.statSync (fs.js:948:11)
    at checkDirectorySync (server/methods.js:94:8)
    at methods.js (server/methods.js:101:1)
    at fileEvaluate (packages\modules-runtime.js:343:9)
    at require (packages\modules-runtime.js:238:16)
    at D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\server\app\app.js:453:1
    at infos.forEach.info (D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\server\boot.js:414:13)
    at Array.forEach (<anonymous>)
    at D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\server\boot.js:413:9
    at D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\server\boot.js:463:5
  errno: -4058,
  code: 'ENOENT',
  syscall: 'stat',
  path: 'D:\\D drive\\Ani\\Learnings\\Meteor\\staticsite\\.meteor\\local\\build\\programs\\private\\xmls' }
C:\Users\animeshsharma\AppData\Local\.meteor\packages\meteor-tool\1.6.0_1\mt-os.windows.x86_64\dev_bundle\server-lib\node_modules\fibers\future.js:280
						throw(ex);

will it be possible to write files in private folder?
and then keep updating those when needed?
will adding/updating files in Private folder restart server each time?

Thanks for your help!!!

robfallows · January 10, 2018, 10:48am

Basically, yes. But note that assets accessed via the Assets API are read-only, but if you use the fs commands, you can write to the files. However, bear in mind my comments about deployment. Deploying to many cloud services (including Galaxy and AWS EC2) provides a transient filesystem. While you can use EBS on EC2 to provide permanent disk storage, there is a further disadvantage of local storage: if you are running multiple instances, each has its own, independent local storage. Hence my suggestion to use S3.

I did test this myself to ensure nothing has changed, and I can create, write and read files and folders under private. I used the async version:

fs.stat(directory, err => {
  console.log(err);
  fs.mkdir(directory, err => {
    console.log(err);
  });
});

which worked as expected. The “test and create” method is not recommended, because it offers no advantages over a simple fs.mkdir or fs.mkdirSync, which will always be atomic.

Note that the private folder is not in your application root, but in the runtime root. In development, that’s D:\D drive\Ani\Learnings\Meteor\staticsite\.meteor\local\build\programs\private as reported in your original post. When you deploy into production, your runtime root is whatever you want it to be.

Yes.

Yes - but note my comments about deployment and multiple instances.

Yes and no

Yes, if you create/update files in the private folder in your app’s root in development^*. No, if you do it in the private folder in your runtime root. There is no file watching in a deployed, production app.

^* Interestingly, I’ve just compared Windows with Linux and the Windows file watcher doesn’t pick up changes in the private folder. I think that’s a bug.

ani1219 · January 10, 2018, 11:40am

@robfallows, a big thanks for your response and elaborated answers to my questions!!! However, I was afraid that my post was too overwhelming to get a response.

I have been struggling on this for two days now. Considering the challenges I am facing vs. help and related stuff available on web, I am thinking to change my approach now. Rather than creating folders and files in Private directory, how about storing these files into MongoDb collection and operate from there?

What do you suggest between writing files to Private vs. storing those in DB?

I did some research on this and found your one post on it. I cloned the git repo (https://github.com/robfallows/ukgrid) and tried to run on my local but those API’s don’t seem to work anymore. What should I do to get that working?

What my app is supposed to do is:

Read a given sitemap xml
Two things:
- If #1 is done for the first time, store the pages’ sitemap xmls
- If app has those pages xml stored, compare those with latest data from #1 to find out changes.
Display the list of updated (new, deleted, changed) pages to user from comparison from #2.

This is the functionality of one phase. In second phase, user can go on to create/update a static version of a given site based upon these XMLs.

robfallows · January 10, 2018, 11:52am

That’s basically GridFS (unless you’re rolling your own). You could look at ostrio:files, which integrates with GridFS among many others. Definitely much better than using the local filesystem

ani1219 · January 10, 2018, 11:58am

Great, thank you.

I did some research on this and found your one post on it. I cloned the git repo (https://github.com/robfallows/ukgrid) and tried to run on my local but those API’s don’t seem to work anymore. What should I do to get that working?

Any clue on this? As the description suggests, your app is doing almost the same what I am looking for - getting external XML data, store that into DB, and then read from DB to display.

robfallows · January 10, 2018, 12:03pm

I’ll take a look. As long as the endpoints it’s calling are still there, I’ll update it.

ani1219 · January 10, 2018, 12:07pm

Thank you so much. I am trying your suggestion for ostrio:files. I will update you with my results.

One more thing, what will it take me to be able to post new items in this forum?

robfallows · January 10, 2018, 12:13pm

New topics? You should be able to do that by now: https://forums.meteor.com/badges/1/basic?username=ani1219