Streaming stdout on exec() and bindEnvironment()?


#1

Hi!
Cashing in more karma for somebody else’s node expertise. So, trying to get rid of the run_nightwatch.sh script, and have got the following working in the nightwatch-framework package. Yay! It works! (Sort of.)

Npm.require("child_process").exec("./node_modules/nightwatch/bin/nightwatch", Meteor.bindEnvironment(function(error, stdout){
  console.log(stdout);
  console.log('[nightwatch-framework] Nightwatch exited.');          
}));
```

Problem is, the nightwatch script is very expensive, in that it can run for 5 or 10 minutes or longer as it parses through thousands of tests.  And the solution I came up with, well, it doesn't write the results out to the console until after everything is done and the child process ends.  

The run_nightwatch.sh script, on the other hand, spits the results out as it process them.  So, if you're a minute into a 5 minute test run, and things start blowing up, you can just ctrl-c the test runner, and exit early.  But with the node exec() function, we're blind until all the results have been processed.  

Are there any node gurus in the audience who might understand the details of child_process and exec() better than I, and know how to get the streaming buffer working across processes?  My expertise is really databases and data science and quality assurance processes.  This is pushing the limits of my knowledge of node.

Thanks!
Abigail

How to handle long server processes
#2

I’ve managed to get it working using the following code snippets. I’m considering writing a package to simplify this process - do you think it’d be useful?

Console = new Mongo.Collection('console');

if (Meteor.isServer) {
    exec = Npm.require('child_process').exec;
    Fiber = Npm.require('fibers');

    _execSync = function(cmd, stdoutHandler, stderrHandler) {
        exec(cmd, Meteor.bindEnvironment(
                function(error, stdout, stderr) {
                    if (stdout != "")
                        stdoutHandler(stdout);
                    if (stderr != "")
                        stderrHandler(stderr);
                }
            )
        );
    }

    Meteor.methods({
        consoleExecSync : function(cmd) {
                _execSync(cmd, consoleInsert, consoleInsert);
        }
    });

    consoleInsert = function(_data) {
        Console.insert({
            timestamp : new Date().getTime(),
            data : _data
        });
    }
}

Basically, the way I do it is I pass a “stdoutHandler” and “stderrHandler” to the _execSync function, which then calls the handlers any time the stdout or stderr is flushed. It’s important to note that some programs do a very poor job of flushing the pipe. If the thing you are exec’ing is a python script, for example, you need to explicitly call sys.stdout.flush() in order for this to work (print does not do it by default).

The consoleInsert function here just takes the data and inserts it into a collection. In my application, I have a “console” that I have a {{#each console}} template in that displays the console documents to the client. This results in pretty close to live output from the exec’d program.

Does that help?


Meteor + Python?
#3

Ooooh! Cleverness! This looks like just the thing I was hoping somebody could help me with! I’m going to give it a try!


#4

I copied and pasted this code from an older project of mine. Now that I look at it, I don’t actually think the Fiber = Npm.require('fibers'); is required. Probably left over from when I banged my head against a wall for an hour trying to get this working…

But hey that’s what forums are for! That way you don’t have to bang your head against the same wall!


#5

Sorry to spam you, but a last note:

If nightwatch is not flushing stdout, you’ll get the same issues as you were having above. A good test to see if the live stdout reading is working is to just put a console.log(_data) in stdoutHandler and stderrHandler and then run the cmd find / -name meteor. I know the find command flushes the pipes correctly (from my own testing) and should give you a decent amount of results to see how it’s doing.


#6

Woo!! Figured it out! The stdoutHandler and stderrHandler got me on the right track! Started thinking about stdoutHandler and methods that act on streaming events, and syntax that doesn’t involve parentheses, and that’s when I remembered stdout.on('data'). Final solution was to use spawn rather than exec. Apparently I didn’t even need the bindEnvironment().

var spawn = Npm.require('child_process').spawn;
var nightwatch = spawn('./node_modules/nightwatch/bin/nightwatch');

nightwatch.stdout.on('data', function(data){
  // data is in hex, and has a line break at the end
  console.log(('' + data).slice(0, -1));
});

nightwatch.on('close', function(code){
  console.log('Nightwatch exited with code ' + code);
});

Thank you much for your help! You saved me many hours of frustration! :slight_smile:


#7

Awesome! Glad I could help! Did you manage to get it interacting with any Meteor APIs like a collection? Or is it just console.log’ing right now? That’s when you may need Meteor.bindEnvironment.