[Solved] Meteor 1.5.2.2 deployment on EBS issue with "fibers/future"

My meteor app runs on v1.5.2 and works great.
Upgraded to v1.5.2.2, still works great locally (macos) but deployed version is throwing an error. Deployment is via aws elastic beanstalk. The ec2 environment logs show that the app is throwing this error:

> node main.js

/var/app/current/programs/server/boot.js:392
}).run();
   ^
Error: Cannot find module "fibers/future"
    at Object.require (/var/app/current/programs/server/boot.js:232:24)
    at packages/meteor.js:93:20
    at packages/meteor.js:263:4
    at packages/meteor.js:1392:3
    at /var/app/current/programs/server/boot.js:339:34
    at Function._.each._.forEach (/var/app/current/node_modules/underscore/underscore.js:153:9)
    at /var/app/current/programs/server/boot.js:158:5
    at /var/app/current/programs/server/boot.js:388:5
    at Function.run (/var/app/current/programs/server/profile.js:510:12)
    at /var/app/current/programs/server/boot.js:387:11

What could have caused that, solutions, ideas? :face_with_raised_eyebrow:

For reference, some interesting facts which may be relevant:
my package.json has:

    "fibers": "^2.0.0",

my build script is running (among other commands):

meteor yarn install
meteor build --directory [my app's dir] --architecture os.linux.x86_64
zip  ... # creating bundle

I run the build script on both macos and Linux.
When deploying the resulting bundle to beanstalk before the v1.5.2.2 (also before v1.5.2.1 ) update, the app started successfully. The update definitely broke it :alarm_clock: :hammer: :hushed:

I am aware there’s a shitstorm involving bcrypt binaries, but app does not include it, so this option gets ruled out…

Could this issue be one of em issues which result from missing the target architecture binary of fibers/future? If so, I have to wonder how does meteor yarn install handle those things. After all, I’m not providing the architecture in the yarn command.

Also, not sure if this matters; a while back ago my app was running on an older meteor version (~v1.1). Since then, I’ve been bumping it up regularly up to v1.5.2. I wonder if there’s anything under .meteor/packages which may be breaking fibers/future. At 1st glance, no obvious culprits…

So, ideas?

I have a deployment script for EVERY new build I put up.

This is the KEY in resolving that issue.

npm install --production

You’re welcome :slight_smile:

Self hosting is a bitccccccccccccch.

NOT HERE:

npm install --production <---- WRONG
meteor yarn install
meteor build --directory [my app’s dir] --architecture os.linux.x86_64
zip … # creating bundle

When SSH’d in to your server:

cd /home/andy/SCO/production/programs/server
npm install --production <----------------- HERE
node main.js

1 Like

@SkyRooms awesome, great tip!

Would you say it’d best go into the package.json script section?
atm now it’s:

  "scripts": {
    "start": "node main.js"
  },

I could make it something like:

  "scripts": {
    "install": "npm install --production",
    "start": "node main.js"
  },

or idk even:

  "scripts": {
    "start": "npm install --production ; node main.js"
  },

wdyt?

1 Like

I’m not sure, I don’t know enough about package.json…

I have on my production server running on Ubuntu, a file called START_SERVER.sh

I run it ./START_SERVER.sh

I essentially re-install EVERYTHING. It’s overkill, but I haven’t had any problems with it for months.

1 Like

www.StarCommanderOnline.com

Edit: got mixed up a bit, thought I got it to work earlier but didn’t :disappointed:

Anyhow, @SkyRooms you definitely got me on the right track :1st_place_medal:

I would like to figure out how to update fibers according to best practices + package.json.

To clarify, aws ebs forces a strict deployment flow. I’d like to stick to that flow since there are benefits involved (security, continuous integration, etc). I’d imagine this flow executes whatever necessary from the bundle’s package.json. Thus I should be able to run npm install cmd somehow from there in order to rebuild fibers. Question is, how.

Q to those of you who manage their own deployment (perhaps using aws/ebs):

How are you updating your node_modules to make fibers healthy?

btw according to npmjs in package.json scripts section there are the start, install and preinstall sections. Does anyone have experience/thoughts about placing the npm install script in one of those?

1 Like

@tivoni I’m fighting the same issue. Adjusting package.json to include the --production flag doesn’t seem to work since it causes the npm install to run recursively.

So I’ve added the environment variable NODE_ENV=production to my EB deployment.

If you find any other solutions, please let me know.

@bliles what does the NODE_ENV change achieve in this context? I don’t follow

Sorry, I thought that the key fix was the “production” setting when running npm install. I later figured out that the main part of the fix was running npm install --production in the programs/server folder. I’m currently testing a new approach using an elastic beanstalk container command to run this npm install:

In my elastic beanstalk deployment bundle/.ebextensions I have added this file I’m calling server-npm-install.config

container_commands:
  server_npm_install:
    command: |
      cd /tmp/deployment/application/programs/server/ && npm install --production

I am getting the following error after adding the configuration you mentioned

ERROR: [Instance: i-03ddc877f131f8e20] Command failed on instance. Return code: 127 Output: /bin/sh: npm: command not found. 
container_command server_npm_install in .ebextensions/npm-install.config failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.

@bliles did you happen to run into a similar issue?

1 Like

Yes, I’m still fighting with this trying to get a working config. I was hoping my attempt here would help someone else possibly get it working so they could share. If I get a working config I will post.

SOLVED!

First I had to create a shell script in the project that is included in my deployment bundle. This script runs npm install --production in the /tmp/deployment/application/programs/server/ folder. I used a shell script because I had a terrible time getting the .ebextension container command to run with the proper environment (path to npm).

server-npm-install.sh

#!/bin/bash
export PATH=$PATH:$NODE_HOME/bin
cd /tmp/deployment/application/programs/server
npm install --production

Next we need an .ebextension file to run this command:

00_app_npm.config

container_commands:
  00_app_npm:
    command: ./server-npm-install.sh
    cwd: /tmp/deployment/application

This approach assumes that the server-npm-install.sh script is in the root of your deployment bundle. The /tmp/deployment/application path is the path where elasticbeanstalk prepares your app version before switching to the new version and restarting node.

I still see the error: cannot run in wd %s %s (wd=%s) meteor-dev-bundle@0.0.0 node npm-rebuild.js /tmp/deployment/application/programs/server, but my app starts up and runs just fine.

Getting Return code: 126 Output: /bin/sh: ./server-npm-install.sh: Permission denied.

Do you have a .config file that sets permissions as well?

@416serg I’m guessing bliles had execute permissions set locally on the shell script before committed in git

@bliles kudos for the solution =)
I cannot but wonder:

  1. how did fibers work until v1.5.2 without needing this workaround? If there’s a way mdg can simply solve it in one of the next updates, it might be worthwhile for me to wait for it
  2. Is there a different, more elegant solution which does not involve the ebextensions brute force method. I’m thinking in a direction of something in package.json
    :nerd_face:

@bliles figured out permissions but still getting npm command not found, now from the shell script :frowning:

@416serg I don’t know why you would get that error, but perhaps the environment where you are running the installation does not have the variable $NODE_HOME populated for some reason? You could test this hypothesis by doing echo $NODE_HOME in the script. If that is the case, you could try updating the EB environment to a newer version. Mine is 64bit Amazon Linux 2017.03 v4.3.0 running Node.js. You could also just find the path to the npm binary you are using, mine is /opt/elasticbeanstalk/node-install/node-v6.11.1-linux-x64/bin/. I used the $NODE_HOME env variable so that the npm used will correspond to the node version specified in the elastic beanstalk config.

1 Like

@tivoni I definitely agree this seems like a regression to me since it was not a problem previously.

Yeah, I neglected to mention that I chmod +x server-npm-install.sh in my repo. It’s second nature to enable the execute bit on a shell script for me so I didn’t even think to mention it.

It’s a little late to the party, but would like to offer my version of overcoming the recent issues involving the elastic-beanstalk deploy.

The container commands section I used:

container_commands:
  meteor-server-npm-install:
    command: |
      echo "meteor fibers fix/hack"
      cd programs/server
      PATH="$PATH:$NODE_HOME/bin" npm install --production --unsafe-perm

Also, note that AWS reps provided me the following file to overcome some libc issues which fail the configuration phase:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh" :
    mode: "000775"
    owner: root
    group: root
    content: |
      #!/bin/bash
      #==============================================================================
      # Copyright 2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
      #
      # Licensed under the Amazon Software License (the "License"). You may not use
      # this file except in compliance with the License. A copy of the License is
      # located at
      #
      # http://aws.amazon.com/asl/
      #
      # or in the "license" file accompanying this file. This file is distributed on
      # an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or
      # implied. See the License for the specific language governing permissions
      # and limitations under the License.
      #============================================================================== 
      set -xe

      sudo /opt/elasticbeanstalk/containerfiles/ebnode.py --action npm-install 

commands:
  remove_old_file:
    command: "rm -fr /opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh.bak"
    ignoreErrors: true

I hope it helps someone as much as it helped me :slight_smile: