Best use of Meteor.bindEnvironment for stream processing

help
1

I have code that imports GTFS data. Request is used to download the archive, which is then piped to unzip.Parse() and then individual files are checked against the database and their data upserted. I’m used to using Meteor.wrapAsync() on a few functions to convert them into synchronous code, but in this instance I think that would be a bad idea because there’s so much data and I’d rather stream it than work with it in huge blocks. I also have this import code partially working outside of Meteor, am fairly confident that it does the job, and would like to retain as much of its structure as possible.

Unfortunately, when it comes time to do the upsert, I get an error telling me to call my code in Meteor.bindEnvironment. I think I do this, the code seems to run and I get logs on the console, but the error never goes away. What am I missing? Here’s my code in LiveScript, hopefully it’s readable enough for folks:

importGtfs: (url) ->
  f = new Future()
  parser = -> csv.parse({columns: true})
  request(url)
  .pipe(unzip.Parse())
  .on "entry", Meteor.bindEnvironment (entry) !->
    console.log("Importing "+entry.path)
    switch entry.path
    | "agency.txt" =>
      p = parser()
      entry.pipe(p)
      .on "readable", !->
        while record = p.read()
          console.log(record)
          Agencies.upsert({agency_name: record.agency_name}, record)
    | "routes.txt" =>
      p = parser()
      entry.pipe(p)
      .on "readable", !->
        while record = p.read()
          console.log(record)
          Routes.upsert({route_id: record.route_id}, record)
    | otherwise => entry.autodrain()
  .on "close", !-> f.return()
  f.wait()
2

BTW, here is the JS output of the LiveScript in case the above is difficult to read:

Meteor.methods({
  importGtfs: function(url){
    var f, parser;
    f = new Future();
    parser = function(){
      return csv.parse({
        columns: true
      });
    };
    request(url).pipe(unzip.Parse()).on("entry", Meteor.bindEnvironment(function(entry){
      var p;
      console.log("Importing " + entry.path);
      switch (entry.path) {
      case "agency.txt":
        p = parser();
        entry.pipe(p).on("readable", function(){
          var record;
          while (record = p.read()) {
            console.log(record);
            Agencies.upsert({
              agency_name: record.agency_name
            }, record);
          }
        });
        break;
      case "routes.txt":
        p = parser();
        entry.pipe(p).on("readable", function(){
          var record;
          while (record = p.read()) {
            console.log(record);
            Routes.upsert({
              route_id: record.route_id
            }, record);
          }
        });
        break;
      default:
        entry.autodrain();
      }
    })).on("close", function(){
      f['return']();
    });
    return f.wait();
  }
});
3

OK, a bit more. I found this which seems to be my issue, but breaking out the bindEnvironment calls doesn’t seem to change anything.

Is there some way I can log what Meteor thinks the environment is to the console so I can find where the environment is getting lost? The only bindEnvironment reference in the docs states that Meteor uses global environment variables. I tried logging process.env but it’s always undefined, even in the root of the method and not in any callbacks.

Thanks.

4

Ugh, stupid human trick. :slight_smile: Got it sorted. In my attempt to flatten the function-calling tree into a set of uniquely-named functions in a scope where bindEnvironment would work, I had named one process. So of course process.env was undefined.

For anyone hitting this problem, it does look like you can view process.env to track down where in the callback spaghetti your environment is getting lost.

Anyhow, all good now.