Meteor + Python?

Hello,

I have a Python script that does some computationally intense crunching and I foresee the need to connect it to my Meteor app later on down the road. There are a lot of code snippets scattered around the internet explaining the process of connecting Meteor to Python but I thought I would ask the pros first (you guys).

So, what is the recommended method for communicating with a Python script from Meteor? Code examples would be super helpful.

Thanks!

4 Likes

Depending on what the data is you could just write it to a json file with python and then read it with meteor. Or connect to the same database url with PyMongo

3 Likes

To actually execute the process, see this post for some code snippets for executing an external script. This is what I have done to execute a python script. That same post shows how to handle the stdout and stderr of the script, so you could pass data that way. I have used this to get a “console”-like streaming results from the executed Python.

To get nice reactive results from the python script, @corvid nailed it on the head with pymongo. Make sure that the pymongo has the C extensions, or it will be horribly slow. See here for more details on that. In my experience, there was a several hundred times speed up when pymongo.has_c() returns true.

A combination of the two methods will allow you to get some nice stdout debug/progress messages and get the useful data from the database updates through pymongo.

If you have any more specific questions, let me know.

3 Likes

@corvid: I am new to Meteor and has exactly same question as @fire_water. We used to use RabbitMQ to make Python programs work with Node.js - In Meteor case, the Python program is most likely on a different server doing machine learning and optimization tasks - how Meteor triggers a Python program on a different server and how Meteor is notified when the Python program completes the tasks? Thanks a lot for your help.

@jchristman , thank you for your help with this. I really appreciate it.

I am still trying to get a basic example working but I am running into some trouble with it. I have included my code below. If you have a moment, would you mind reviewing my code and letting me know what I did wrong?

Basically, the app should simply display the result of the Python script which, in this case, is “Hello from Python script!”. Unfortunately, that doesn’t seem to be working and I have no clue why since both Terminal and the web console are not displaying any error/warning messages:

JS

if (Meteor.isClient) {

	Template.seq_opt_calc.helpers({
		message: function () {
			return Session.get("message");
		},
		message_defined: function () {
			return Session.equals("message", undefined) ? false : true;
		}
	});

	Template.seq_opt_calc.events({
		"submit #seq_form": function (event) {
			// This function is called when the form is submitted.

			event.preventDefault();  // Prevent page reload.

			var form_data = {
				seq: event.target.seq.value
			};

			if (!Match.test(form_data.seq, String)) {
				Session.set("message", "Please enter a sequence.");
				return false;
			}

			if (form_data.seq.length < 3) {
				Session.set("message", "Sequence must be at least 3 characters");
				return false;
			}

			Meteor.call("consoleExecSync",form_data.seq,function(error, result){
				if (error) {
					alert(error);
					//Session.set("message", JSON.stringify(error));
					Session.set("message", error.reason);
					return false;
				}
				if (result) {
					alert(result);
					Session.set("message", result);
					return false;
				}
			});
		}
	});
}

if (Meteor.isServer) {

	exec = Npm.require('child_process').exec;

	Meteor.methods({
		consoleExecSync : function(seq) {

			var cmd = "python /Users/administrator/Desktop/test.py";

			exec(cmd, Meteor.bindEnvironment(
				function(error, stdout, stderr) {
					if (error) {
						throw new Meteor.Error(error, error);
					}
					if (stdout) {
						return stdout;
					}
					if (stderr) {
						return stderr;
					}
				}
			));
		}
	});
}

Python

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from __future__ import print_function
import sys

def main():

	print("Hello from Python script!", file=sys.stdout)
	sys.stdout.flush()

if __name__ == '__main__':
	main()

@fire_water, right off the bat, I see you are trying to return the stdout and stderr from the function in the exec function. I suppose that the name of the function _execSync is deceiving in that it seems as if you should be able to return the stdout or stderr. This is, however, not the case.

In fact, the function inside of Meteor.bindEnvironment is called each time stdout or stderr flushes. That is, after sys.stdout.flush(), the function inside of the exec call will be called with stdout as “Hello from Python script!”. The way that I capture this data is by inserting it into a collection inside of that function to which the clients are subscribed. I would change your code to something like this:

// Define the collections that are on the client and server
Stdout = new Mongo.Collection('stdout');
Stderr = new Mongo.Collection('stderr');

if (Meteor.isClient) {
    Meteor.subscribe('stdout');
    Meteor.subscribe('stderr');
}

if (Meteor.isServer) {
        Meteor.publish('stdout', function() { return Stdout.find(); } );
        Meteor.publish('stderr', function() { return Stderr.find(); } );

	exec = Npm.require('child_process').exec;

	Meteor.methods({
		consoleExecSync : function(seq) {

			var cmd = "python /Users/administrator/Desktop/test.py";

			exec(cmd, Meteor.bindEnvironment(
				function(error, stdout, stderr) {
					if (error) {
						throw new Meteor.Error(error, error);
					}
					if (stdout) {
                                            Stdout.insert({
                                                timestamp : new Date().getTime(),
                                                data : stdout
                                            });
					}
					if (stderr) {
                                            Stderr.insert({
                                                timestamp : new Date().getTime(),
                                                data : stderr
                                            });
					}
				}
			));
		}
	});
}

If that still doesn’t work, try console.log’ing the stdout and stderr and see if it you can see the results on the server. If you can’t, then the stdout is not flushing correctly. I haven’t tested your code, but I usually do the following to write to sys.stdout in python.

import sys

sys.stdout.write('Hello from Python script!')
sys.stdout.flush()

Hope that helps!

1 Like

@jchristman Hi. I have been trying to use your code to bring log info from server backend to webpage. Mostly it works. But I only got 1 stderr in the end, which is not what I want. We want it real-time, so should be several stderr coming from python execution. I am rather new to Meteor and am working on some old code, wondering if its the future that changed the behavior? The return is problematic without future. and even in that case, I still get one stderr.

let exec = Npm.require(‘child_process’).exec;
let Fiber = Npm.require(‘fibers’);
let Future = Npm.require(‘fibers/future’);

Meteor.methods({
	test_consoleExecSync: function(options) {
	    Stderr.remove({});
	    let future = new Future()
        let arg = JSON.stringify(options);

		var cmd = 'python3 test.py' ;

		exec(cmd, Meteor.bindEnvironment(
			function(error, stdout, stderr) {
				if (error) {
					throw new Meteor.Error(error, error);
				}

				if (stderr) {           console.log("stderr:", stderr);
                                        Stderr.insert({
                                            timestamp : new Date().getTime(),
                                            data : stderr
                                        });
				}

				if (stdout) {
                        console.log("stdout:", stdout);
                        var data = JSON.parse(JSON.stringify(stdout));
                       return future.return(data);
				}

			}
		));

		        // Wait and return
    try {
        let wait = future.wait()
        return future.wait();
    }
    catch(err) {
        console.error("Server error: ", err);

    }

	}
});

BTW, I added several sys.stderr.flush() in python code as I saw your other answer. So really wondering if its python problem or js problem.

To be completely honest I have no idea if this still works in the most recent builds of Meteor. These posts are three years old and several major versions of Meteor old. I can take a look tonight and see what I can figure out, but you may be better off looking for a more recent implementation of this. I’m certain that there are better ways of doing this nowadays.

Please post a link here if you find a better way to do it!!

:slightly_smiling_face:

Now we can do it this way:
progress = exec(cmd);
progress.stderr.on(‘data’, Meteor.bindEnvironment(function(stderr){
console.log(“stderr:”, stderr);
Stderr.insert({
timestamp : new Date().toLocaleTimeString(),
data : stderr.toString()});
}));
progress.stdout.on();