Can't use sharp image processing library on Galaxy [SOLVED]

mdorn · March 21, 2017, 5:02pm

I developed a bunch of new functionality for my app that depends on the Sharp image processing library, but when I attempt to deploy to Galaxy, the build process chokes on the node-gyp compilation related steps required for Sharp. I’ve pasted all of the relevant log output below, but the first relevant line seems to be:

g++: error: ../vendor/lib/libvips-cpp.so: No such file or directory

This installation documentation seems relevant, but not sure if there’s any way I can make use of any info there on Galaxy: http://sharp.dimens.io/en/stable/install/#linux

FWIW I can build my app using meteor build without a problem in my OSX development environment.

I’ve already put in a support request to Galaxy, but since they’ve been pretty slow to respond lately, I thought I’d try here. Anyone have any insight?

Relevant log output below:

v093
2017-03-20 21:42:57-05:00Building version 93.
v093
2017-03-20 21:42:57-05:00Pulling base image.
v093
2017-03-20 21:42:57-05:00Step 1 : FROM meteor/galaxy-app@sha256:399572dee64f909fff88db6ea9aafbf13b1070a708f345b53bbbfa21bbfb5b5b
v093
2017-03-20 21:42:57-05:00 ---> 7fc7aff21a31
v093
2017-03-20 21:42:57-05:00Step 2 : RUN /bin/bash -x /app/setup.sh https://prod-galaxy-bundles.s3.amazonaws.com/bundles/ouZnruJ7daXhH8ZkQ/20170321-024022/bundle.gAGurkiRDu2Kr8HmJ.tgz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ4JU4KHU7CAVLZ3Q%2F20170321%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170321T024257Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=1ef9290a3febb73990e4baff0070e196d3d211c7b163c4753007f781387b946a
v093
2017-03-20 21:42:57-05:00 ---> [Warning] Your kernel does not support swap limit capabilities, memory limited without swap.
v093
2017-03-20 21:42:57-05:00 ---> Running in 13d6d2b0f9d0
v093
2017-03-20 21:42:57-05:00+ set -ex
v093
2017-03-20 21:42:57-05:00+ cd /app
v093
2017-03-20 21:42:57-05:00+ tar xz -m
v093
2017-03-20 21:42:57-05:00+ curl -sS 'https://prod-galaxy-bundles.s3.amazonaws.com/bundles/ouZnruJ7daXhH8ZkQ/20170321-024022/bundle.gAGurkiRDu2Kr8HmJ.tgz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ4JU4KHU7CAVLZ3Q%2F20170321%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170321T024257Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=1ef9290a3febb73990e4baff0070e196d3d211c7b163c4753007f781387b946a'
v093
2017-03-20 21:43:00-05:00+ cd /app/bundle
v093
2017-03-20 21:43:00-05:00+ export NODE_VERSION
v093
2017-03-20 21:43:00-05:00++ /app/select_node_version.sh
v093
2017-03-20 21:43:00-05:00+ NODE_VERSION=4.7.3
v093
2017-03-20 21:43:00-05:00+ /app/install_node.sh
v093
2017-03-20 21:43:00-05:00+ curl -SLO https://nodejs.org/dist/v4.7.3/node-v4.7.3-linux-x64.tar.gz
v093
2017-03-20 21:43:00-05:00Installing node version 4.7.3 to /node-v4.7.3-linux-x64
v093
2017-03-20 21:43:00-05:00 % Total % Received % Xferd Average Speed Time Time Time Current
v093
2017-03-20 21:43:00-05:00 Dload Upload Total Spent Left Speed
v093
2017-03-20 21:43:01-05:00
 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 11.6M 100 11.6M 0 0 45.4M 0 --:--:-- --:--:-- --:--:-- 45.5M
v093
2017-03-20 21:43:01-05:00+ tar -xzf node-v4.7.3-linux-x64.tar.gz -C /
v093
2017-03-20 21:43:01-05:00+ rm node-v4.7.3-linux-x64.tar.gz
v093
2017-03-20 21:43:01-05:00+ export PATH=/node-v4.7.3-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
v093
2017-03-20 21:43:01-05:00+ PATH=/node-v4.7.3-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
v093
2017-03-20 21:43:01-05:00+ npm -g install npm@latest
v093
2017-03-20 21:43:08-05:00/node-v4.7.3-linux-x64/bin/npm -> /node-v4.7.3-linux-x64/lib/node_modules/npm/bin/npm-cli.js
v093
2017-03-20 21:43:09-05:00npm@4.4.1 /node-v4.7.3-linux-x64/lib/node_modules/npm
v093
2017-03-20 21:43:09-05:00+ pushd programs/server
v093
2017-03-20 21:43:09-05:00/app/bundle/programs/server /app/bundle
v093
2017-03-20 21:43:09-05:00+ npm install --unsafe-perm
v093
2017-03-20 21:43:11-05:00
v093
2017-03-20 21:43:11-05:00> fibers@1.0.15 install /app/bundle/programs/server/node_modules/fibers
v093
2017-03-20 21:43:11-05:00> node build.js || nodejs build.js
v093
2017-03-20 21:43:11-05:00
v093
2017-03-20 21:43:11-05:00`linux-x64-46` exists; testing
v093
2017-03-20 21:43:11-05:00Binary is fine; exiting
v093
2017-03-20 21:43:11-05:00
v093
2017-03-20 21:43:11-05:00> meteor-dev-bundle@0.0.0 install /app/bundle/programs/server
v093
2017-03-20 21:43:11-05:00> node npm-rebuild.js
v093
2017-03-20 21:43:11-05:00
v093
2017-03-20 21:43:14-05:00
v093
2017-03-20 21:43:14-05:00> bcrypt@0.8.7 install /app/bundle/programs/server/npm/node_modules/bcrypt
v093
2017-03-20 21:43:14-05:00> node-gyp rebuild
v093
2017-03-20 21:43:14-05:00
v093
2017-03-20 21:43:15-05:00make: Entering directory `/app/bundle/programs/server/npm/node_modules/bcrypt/build'
v093
2017-03-20 21:43:15-05:00 CXX(target) Release/obj.target/bcrypt_lib/src/blowfish.o
v093
2017-03-20 21:43:15-05:00 CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt.o
v093
2017-03-20 21:43:15-05:00 CXX(target) Release/obj.target/bcrypt_lib/src/bcrypt_node.o
v093
2017-03-20 21:43:16-05:00 SOLINK_MODULE(target) Release/obj.target/bcrypt_lib.node
v093
2017-03-20 21:43:16-05:00 COPY Release/bcrypt_lib.node
v093
2017-03-20 21:43:16-05:00make: Leaving directory `/app/bundle/programs/server/npm/node_modules/bcrypt/build'
v093
2017-03-20 21:43:19-05:00
v093
2017-03-20 21:43:19-05:00> sharp@0.17.2 install /app/bundle/programs/server/npm/node_modules/sharp
v093
2017-03-20 21:43:19-05:00> node-gyp rebuild
v093
2017-03-20 21:43:19-05:00
v093
2017-03-20 21:43:20-05:00make: Entering directory `/app/bundle/programs/server/npm/node_modules/sharp/build'
v093
2017-03-20 21:43:20-05:00 TOUCH Release/obj.target/libvips-cpp.stamp
v093
2017-03-20 21:43:20-05:00 CXX(target) Release/obj.target/sharp/src/common.o
v093
2017-03-20 21:43:21-05:00 CXX(target) Release/obj.target/sharp/src/metadata.o
v093
2017-03-20 21:43:22-05:00 CXX(target) Release/obj.target/sharp/src/operations.o
v093
2017-03-20 21:43:24-05:00 CXX(target) Release/obj.target/sharp/src/pipeline.o
v093
2017-03-20 21:43:28-05:00 CXX(target) Release/obj.target/sharp/src/sharp.o
v093
2017-03-20 21:43:29-05:00 CXX(target) Release/obj.target/sharp/src/utilities.o
v093
2017-03-20 21:43:30-05:00 SOLINK_MODULE(target) Release/obj.target/sharp.node
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libvips-cpp.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libvips.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libglib-2.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libgobject-2.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libcairo.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libcroco-0.6.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libexif.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libffi.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libfontconfig.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libfreetype.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libgdk_pixbuf-2.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libgif.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libgio-2.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libgmodule-2.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libgsf-1.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libgthread-2.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libharfbuzz.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libjpeg.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/liblcms2.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/liborc-0.4.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libpango-1.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libpangocairo-1.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libpangoft2-1.0.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libpixman-1.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libpng.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/librsvg-2.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libtiff.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libwebp.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libxml2.so: No such file or directory
v093
2017-03-20 21:43:30-05:00g++: error: ../vendor/lib/libz.so: No such file or directory
v093
2017-03-20 21:43:30-05:00make: *** [Release/obj.target/sharp.node] Error 1
v093
2017-03-20 21:43:30-05:00make: Leaving directory `/app/bundle/programs/server/npm/node_modules/sharp/build'
v093
2017-03-20 21:43:30-05:00gyp ERR! build error
v093
2017-03-20 21:43:30-05:00gyp ERR! stack Error: `make` failed with exit code: 2
v093
2017-03-20 21:43:30-05:00gyp ERR! stack at ChildProcess.onExit (/node-v4.7.3-linux-x64/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
v093
2017-03-20 21:43:30-05:00gyp ERR! stack at emitTwo (events.js:87:13)
v093
2017-03-20 21:43:30-05:00gyp ERR! stack at ChildProcess.emit (events.js:172:7)
v093
2017-03-20 21:43:30-05:00gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:211:12)
v093
2017-03-20 21:43:30-05:00gyp ERR! System Linux 4.4.0-57-generic
v093
2017-03-20 21:43:30-05:00gyp ERR! command "/node-v4.7.3-linux-x64/bin/node" "/node-v4.7.3-linux-x64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
v093
2017-03-20 21:43:30-05:00gyp ERR! cwd /app/bundle/programs/server/npm/node_modules/sharp
v093
2017-03-20 21:43:30-05:00gyp ERR! node -v v4.7.3
v093
2017-03-20 21:43:30-05:00gyp ERR! node-gyp -v v3.5.0
v093
2017-03-20 21:43:30-05:00gyp ERR! not ok
v093
2017-03-20 21:43:30-05:00npm ERR! code ELIFECYCLE
v093
2017-03-20 21:43:30-05:00npm ERR! errno 1
v093
2017-03-20 21:43:30-05:00npm ERR! sharp@0.17.2 install: `node-gyp rebuild`
v093
2017-03-20 21:43:30-05:00npm ERR! Exit status 1
v093
2017-03-20 21:43:30-05:00npm ERR!
v093
2017-03-20 21:43:30-05:00npm ERR! Failed at the sharp@0.17.2 install script 'node-gyp rebuild'.
v093
2017-03-20 21:43:30-05:00npm ERR! Make sure you have the latest version of node.js and npm installed.
v093
2017-03-20 21:43:30-05:00npm ERR! If you do, this is most likely a problem with the sharp package,
v093
2017-03-20 21:43:30-05:00npm ERR! not with npm itself.
v093
2017-03-20 21:43:30-05:00npm ERR! Tell the author that this fails on your system:
v093
2017-03-20 21:43:30-05:00npm ERR! node-gyp rebuild
v093
2017-03-20 21:43:30-05:00npm ERR! You can get information on how to open an issue for this project with:
v093
2017-03-20 21:43:30-05:00npm ERR! npm bugs sharp
v093
2017-03-20 21:43:30-05:00npm ERR! Or if that isn't available, you can get their info via:
v093
2017-03-20 21:43:30-05:00npm ERR! npm owner ls sharp
v093
2017-03-20 21:43:30-05:00npm ERR! There is likely additional logging output above.
v093
2017-03-20 21:43:30-05:00
v093
2017-03-20 21:43:30-05:00npm ERR! A complete log of this run can be found in:
v093
2017-03-20 21:43:30-05:00npm ERR! /root/.npm/_logs/2017-03-21T02_43_30_519Z-debug.log
v093
2017-03-20 21:43:30-05:00npm WARN meteor-dev-bundle@0.0.0 No description
v093
2017-03-20 21:43:30-05:00npm WARN meteor-dev-bundle@0.0.0 No repository field.
v093
2017-03-20 21:43:30-05:00npm WARN meteor-dev-bundle@0.0.0 No license field.
v093
2017-03-20 21:43:30-05:00npm ERR! code ELIFECYCLE
v093
2017-03-20 21:43:30-05:00npm ERR! errno 1
v093
2017-03-20 21:43:30-05:00npm ERR! meteor-dev-bundle@0.0.0 install: `node npm-rebuild.js`
v093
2017-03-20 21:43:30-05:00npm ERR! Exit status 1
v093
2017-03-20 21:43:30-05:00npm ERR!
v093
2017-03-20 21:43:30-05:00npm ERR! Failed at the meteor-dev-bundle@0.0.0 install script 'node npm-rebuild.js'.
v093
2017-03-20 21:43:30-05:00npm ERR! Make sure you have the latest version of node.js and npm installed.
v093
2017-03-20 21:43:30-05:00npm ERR! If you do, this is most likely a problem with the meteor-dev-bundle package,
v093
2017-03-20 21:43:30-05:00npm ERR! not with npm itself.
v093
2017-03-20 21:43:30-05:00npm ERR! Tell the author that this fails on your system:
v093
2017-03-20 21:43:30-05:00npm ERR! node npm-rebuild.js
v093
2017-03-20 21:43:30-05:00npm ERR! You can get information on how to open an issue for this project with:
v093
2017-03-20 21:43:30-05:00npm ERR! npm bugs meteor-dev-bundle
v093
2017-03-20 21:43:30-05:00npm ERR! Or if that isn't available, you can get their info via:
v093
2017-03-20 21:43:30-05:00npm ERR! npm owner ls meteor-dev-bundle
v093
2017-03-20 21:43:30-05:00npm ERR! There is likely additional logging output above.
v093
2017-03-20 21:43:30-05:00
v093
2017-03-20 21:43:30-05:00npm ERR! A complete log of this run can be found in:
v093
2017-03-20 21:43:30-05:00npm ERR! /root/.npm/_logs/2017-03-21T02_43_30_552Z-debug.log
v093
2017-03-20 21:43:31-05:00Removing intermediate container 13d6d2b0f9d0
v093
2017-03-20 21:43:31-05:00Failed to build version 93. Retry scheduled at 21 Mar 17 02:43 UTC.

mdorn · March 21, 2017, 11:56pm

Received a response from Galaxy support:

It looks like sharp has a special part of its build script which downloads libraries for the platform you’re running on when you install it. Unfortunately, it doesn’t seem clever enough to re-download the libraries when you run npm rebuild on another platform, which is what Galaxy does. When you upload your built app it contains Mac libraries that won’t work on Linux.

Options you could try include:

Deploying from Linux instead of Mac (eg, from a CI system, a VM on your machine, a VM in the cloud…)

We are hoping to release “custom base image” support for Galaxy very soon, at which point you can install the native libraries globally on your container, which might work.

Working with the Sharp module’s maintainers to see if they can change it to properly work with npm rebuild.

I’ll update back here and mark as solved if deploying from Linux solves the problem.

mdorn · March 22, 2017, 1:55am

Deploying from Linux instead of OS X as per Galaxy support staff’s suggestion solved the issue.

themuffinman · April 6, 2018, 3:10pm

Seems like a kludgy way to operate. Any updates to this?

a.com · April 7, 2018, 11:45pm

I think that I ended up creating a AWS lambda function instead