AWS OpsWorks, Meteor, Docker Deployment

@adamgins Looks like your web socket connection and your http connection to your client are not the same - while one is being loaded over SSL, the websocket connection does not look like it is being supplied over a secure connection as well.

@patrickwml curl: (6) Couldn't resolve host 'install.meteor.com' - looks like when you were trying to build your image, your machine could not connect to the meteor install servers.

@jkatzen thanks.

Not sure if you saw my update, I had rolled back to cluster 1.6.4 and it now works I logged it here https://github.com/meteorhacks/cluster/issues/67

I thought the internal connections did not have to be HTTPs, thatā€™s how itā€™s been running the whole time (before I upgraded to cluster 1.6.7) - is that an issue? I assume to fix this I would need to add each node to my DNS and use a domain that would work with SSL, is that correct?

So still a bit unclear if this is for me to fix or for cluster?

@jkatzen For some reason It wouldnt connect at all so I deleted all containers and images and started over. Once I did so I was able to move forward, however now this is what I see:

bash-3.2$ docker build -t baceapp/app .
Sending build context to Docker daemon 9.886 MB
Sending build context to Docker daemon 
Step 0 : FROM meteorhacks/meteord
# Executing 2 build triggers
Trigger 0, COPY ./ /app
Step 0 : COPY ./ /app
Trigger 1, RUN bash $METEORD_DIR/on_build.sh
Step 0 : RUN bash $METEORD_DIR/on_build.sh
 ---> Running in 06c85feb1453
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6121    0  6121    0     0   3985      0 --:--:--  0:00:01 --:--:-- 17897
Downloading Meteor distribution
######################################################################## 100.0%

Meteor 1.1.0.2 has been installed in your home directory (~/.meteor).
Writing a launcher script to /usr/local/bin/meteor for your convenience.

To get started fast:

  $ meteor create ~/my_cool_app
  $ cd ~/my_cool_app
  $ meteor

Or see the docs at:

  docs.meteor.com 

Sadly it doesnā€™t do anything else, is this normal? Im new to docker so i have no idea if this is what I should see of if it should be running my app, and if it is running my app idk how to access it. Please bare with me, iā€™m just a little lost.

@patrickwml It might seem like itā€™s hanging, but itā€™s actually working in the background - if you leave it running, eventually it will proceed to the next step (which will be installing npm packages which may make it seem like itā€™s hanging again)

@jkatzen Yeh it must have been. Now when I run the build command it finishes almost instantly everything seems to be good now. Thanks for your help!

@jkatzen can you explain this line of the docker file to me COPY /path/to/meteor_settings/file /path/to/container/settings/file So from what iv read COPY works like so COPY <src> <dest> where dest is on that container. However I have no idea where /path/to/container/settings/file lives on the container how can I find this path?

I recently started having problems with private docker repos. When I put in my registry_username and registry_password as my ENV variables in my App on Opsworks, I get the following error. Anyone know why that may be happening?

docker_registry[] (owdocker::docker-image-deploy line 48) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ā€˜1ā€™
---- Begin output of docker login --password=ā€œpasswordā€ --username=ā€œusernameā€ ----
STDOUT: Email: EOF

@patrickwml the COPY line takes a file from your host machine (the machine you are building the docker image on) and puts it inside the container image you are building. So /path/to/container/settings/file is wherever you put your settings file.

@arjunrajjain are you using stock recipes? Is your registry_email environment variable set?

2 Likes

oh whoops, that was the one env variable I was missing haha. Otherwise, after fixing that, iā€™ve been getting errors regarding ā€œno space left on deviceā€ when I setup my instances now. Iā€™m trying to get the failure logs, but Opsworks keeps on saying ACCESS DENIED when I try to download themā€¦

^ Iā€™m getting this error over and over again whenever my instances tried to pull from dockerā€¦it succeeds in pulling some of the images, but then it fails right at the end. Itā€™s weird thoughā€¦during the build process on my computer, there doesnā€™t seem to be anything excessive being bundled. In fact this meteor project is even smaller than one Iā€™ve previously uploaded on Opsworksā€¦any ideas?

Edit -
Ok going a bit further into thisā€¦it looks like redeployments are working fineeā€¦but owdocker::install is failing

Hi @arjunrajjain I have been having some issues (getting 404ā€™s and 500ā€™s on my docker images). Itā€™s almost random on when this happens. I reached out to Docker Support and got this response:

Are you using a version of Docker that is less than 1.6?

If so could you upgrade and try again?

There have been some major changes on our backend that require upgrading to Docker versions greater than 1.6. (Weā€™re migrating from V1 to V2)

I am using Nitrous Pro and ensured that I am running on 1.6
I cleaned up old images / containers with

docker rmi -f `docker images -aq` 

and the rebuilt the container, pushed to dockerhubā€¦ It worked ā€¦ but I am not sure if it was fluke or not.

Ah just to addā€¦ changed the stack settings to:

the Chef Version is 11.10
Berkshelf version3.2.0
OpsWorks Agent version3422 (Jun 29th 2015) (Changelog)

@adamgins - so I delved a bit deeper into this. and two things

  1. Redeployments with docker are working perfectly fine
  2. Setting up an instances fails on the ā€œSetupā€ scripts (owdocker::install)

@khamoud @jkatzen both @arjunrajjain and I are hitting this issue where we cannot start a new instance.

We get an error stating the instance setting has fail, but there are no logs files.

After a quick look around it seems like thereā€™s no disk space.

The instance was started with EBS backed option. If I ssh into the box the space for /dev/xvda1 show as 97% used on a 78GB disk. @arjunrajjain was getting 100% or 99% used. Could there be an issue with allocating space on the image while it installs? This would also explain why there is not enough space for log files and the setup fails.(I am just guessing here).

Any ideas pls?

@adamgins @arjunrajjain

try cleaning up the instance of docker images:

remove images: docker rmi $(docker images -qf "dangling=true")
kill containers and remove them: docker rm $(docker kill $(docker ps -aq))
note: might need sudo

Sorry I havenā€™t been that responsive. Iā€™m currently working on a new deployment with ECS that will hopefully be way more reliable and much easier to set up since it would not have to deal with chef recipes at all. All config

@jkatzen thanks we did do a clean of the docker images (with docker rmi command above) ā€¦ but thatā€™s only on the build machine, right? ie this issue is on the new instance as itā€™s starting upā€¦ or are yo saying ssh into thar and run these commands?

ps keen to see what happens with ECS :slight_smile:

@adamgins: Oh thatā€™s interesting - perhaps when using opsworks, is it possible that it is attaching the same drive at /dev/xvda1 for all of them? If so, then will need a way to clean up the shared space

@jkatzen is just seems to have assigned an 8GB disk. I could try and re-attach a larger one and re-run the setup.

I checked all the other ec2 instances and they are all using different ebs volumes, if thatā€™s what you meant?

Delving a bit deeper - @adamgins @jkatzen
I sshed into the container and it seems like the meteor app is running fine, but I donā€™t know why it says the setup failed :

Hi @arjunrajjain, how goes? Did you manage to get any further with this?

@arjunrajjain how goes? Any progress?
@khamoud @jkatzen did you manage to get anywhere on the AWS alternative?