AWS OpsWorks, Meteor, Docker Deployment


#71

I recently started having problems with private docker repos. When I put in my registry_username and registry_password as my ENV variables in my App on Opsworks, I get the following error. Anyone know why that may be happening?

docker_registry[] (owdocker::docker-image-deploy line 48) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of docker login --password=“password” --username=“username” ----
STDOUT: Email: EOF


#72

@patrickwml the COPY line takes a file from your host machine (the machine you are building the docker image on) and puts it inside the container image you are building. So /path/to/container/settings/file is wherever you put your settings file.

@arjunrajjain are you using stock recipes? Is your registry_email environment variable set?


#73

oh whoops, that was the one env variable I was missing haha. Otherwise, after fixing that, i’ve been getting errors regarding “no space left on device” when I setup my instances now. I’m trying to get the failure logs, but Opsworks keeps on saying ACCESS DENIED when I try to download them…


#74

^ I’m getting this error over and over again whenever my instances tried to pull from docker…it succeeds in pulling some of the images, but then it fails right at the end. It’s weird though…during the build process on my computer, there doesn’t seem to be anything excessive being bundled. In fact this meteor project is even smaller than one I’ve previously uploaded on Opsworks…any ideas?

Edit -
Ok going a bit further into this…it looks like redeployments are working finee…but owdocker::install is failing


#75

Hi @arjunrajjain I have been having some issues (getting 404’s and 500’s on my docker images). It’s almost random on when this happens. I reached out to Docker Support and got this response:

Are you using a version of Docker that is less than 1.6?

If so could you upgrade and try again?

There have been some major changes on our backend that require upgrading to Docker versions greater than 1.6. (We’re migrating from V1 to V2)

I am using Nitrous Pro and ensured that I am running on 1.6
I cleaned up old images / containers with

docker rmi -f `docker images -aq` 

and the rebuilt the container, pushed to dockerhub… It worked … but I am not sure if it was fluke or not.

Ah just to add… changed the stack settings to:

the Chef Version is 11.10
Berkshelf version3.2.0
OpsWorks Agent version3422 (Jun 29th 2015) (Changelog)


#76

@adamgins - so I delved a bit deeper into this. and two things

  1. Redeployments with docker are working perfectly fine
  2. Setting up an instances fails on the “Setup” scripts (owdocker::install)

#77

@khamoud @jkatzen both @arjunrajjain and I are hitting this issue where we cannot start a new instance.

We get an error stating the instance setting has fail, but there are no logs files.

After a quick look around it seems like there’s no disk space.

The instance was started with EBS backed option. If I ssh into the box the space for /dev/xvda1 show as 97% used on a 78GB disk. @arjunrajjain was getting 100% or 99% used. Could there be an issue with allocating space on the image while it installs? This would also explain why there is not enough space for log files and the setup fails.(I am just guessing here).

Any ideas pls?


#78

@adamgins @arjunrajjain

try cleaning up the instance of docker images:

remove images: docker rmi $(docker images -qf "dangling=true")
kill containers and remove them: docker rm $(docker kill $(docker ps -aq))
note: might need sudo

Sorry I haven’t been that responsive. I’m currently working on a new deployment with ECS that will hopefully be way more reliable and much easier to set up since it would not have to deal with chef recipes at all. All config


#79

@jkatzen thanks we did do a clean of the docker images (with docker rmi command above) … but that’s only on the build machine, right? ie this issue is on the new instance as it’s starting up… or are yo saying ssh into thar and run these commands?

ps keen to see what happens with ECS :slight_smile:


#80

@adamgins: Oh that’s interesting - perhaps when using opsworks, is it possible that it is attaching the same drive at /dev/xvda1 for all of them? If so, then will need a way to clean up the shared space


#81

@jkatzen is just seems to have assigned an 8GB disk. I could try and re-attach a larger one and re-run the setup.

I checked all the other ec2 instances and they are all using different ebs volumes, if that’s what you meant?


#82

Delving a bit deeper - @adamgins @jkatzen
I sshed into the container and it seems like the meteor app is running fine, but I don’t know why it says the setup failed :


#83

Hi @arjunrajjain, how goes? Did you manage to get any further with this?


#84

@arjunrajjain how goes? Any progress?
@khamoud @jkatzen did you manage to get anywhere on the AWS alternative?


#85

@arjunrajjain @jkatzen how goes? Are you having any issues with Docker and AWS lately… I just started getting the following error:


---- Begin output of /opt/aws/opsworks/local/bin/berks vendor /opt/aws/opsworks/current/berkshelf-cookbooks ----
STDOUT: Resolving cookbook dependencies...
Fetching cookbook index from https://supermarket.getchef.com...
Installing apt (2.7.0)
Installing aufs (0.1.1)
E, [2015-08-13T09:57:01.408737 #19121] ERROR -- : Actor crashed!
Errno::ETIMEDOUT: Connection timed out - connect(2)

Any help appreciated


#86

Mmm… looks like something broken with chef supermarket here https://forums.aws.amazon.com/thread.jspa?messageID=667893


#87

Back up and running now http://status.chef.io/post/126583232861/resolved-supermarket-berkshelf-failures


#88

I’ve since switched back to using Mupx…spent way too many hours trying to debug Opsworks :frowning:


#89

Anyone run into issues with IE? When running locally, IE loads my site just fine. However, after deploying I get a blank screen for IE. Other browsers are fine.


#90

Hey there,

Have been using this setup (with a few local mods) for some time now. However, it seems it depends on the chef-docker project (https://github.com/bflad/chef-docker) which has since abandoned support for chef < 12. So, when I redeploy or restart existing servers, I don’t seem to have issues. But when I start a new one from scratch, I get errors.

Anyone else run into this? Am I just doing something wrong? Thanks for any tips.