Docker Docker Docker

[migurski]

This in-progress PR will address issue #562, and after merge will likely mark version 5.0.0 of OA Machine.

• Update Circle and Travis CI configurations to use Docker instead of Chef
• Merge @iandees changes from PR Adding my attempt at running tests in Docker #547, and ensure that @jalessio suggestions are being used
• Merge @MiniCodeMonkey changes from PR Add Docker support #159, and adapt README changes with local Docker use documentation
• Modify EC2 userdata scripts to use Docker instead of Chef
• Modify Docker Hub images to work for testing and deployment

These two recent CI builds for commit 251a194 demonstrate working use of Docker:

• https://travis-ci.org/openaddresses/machine/builds/214842821
• https://circleci.com/gh/openaddresses/machine/1206

[trescube]

Hey, sorry, this fell off my radar. I tested this tonight and followed what I gathered are the instructions:

1. ran docker pull openaddr/prereqs || true
2. ran docker build -f Dockerfile-prereqs -t openaddr/prereqs:latest .
3. ran docker build -f Dockerfile-testing -t openaddr/testing:latest .
4. ran docker-compose up -d && sleep 15 (apparently postgres needs a little time)
5. ran docker-compose run machine python3 /usr/local/src/openaddr/test.py

Here's the output of the last command:

[stephenhess@slumlord machine (migurski/docker-docker-docker)]$ docker-compose run machine python3 /usr/local/src/openaddr/test.py
.............................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 269 tests in 70.808s

OK

That all looks great! Now, what do I do with it? Can I submit sources to the local machine or do machine dev work?

[migurski]

Thanks @trescube, that’s perfect! In theory, you should be able to do both. I’m not 100% clear on how to best configure or recommend volume mapping as with the Vagrant setup, but that would be the right thing to set up. Could I talk to you about this sometime?

[trescube]

Absolutely, you're west coast and I'm east coast, so pretty much any time after 1:30pm eastern works for me. It's pretty easy to use on vagrant now but any way to make it even easier is great with me. Just hit me up on slack! I'm off on Friday but available on Thursday (3/13).

[migurski]

Okay — I'm running from thing to thing tomorrow, but I might be able to carve out some time to hang out and chat.

[migurski]

I rewrote the Vagrant portions of the installation docs to reflect possible new Docker behavior:

• https://github.com/openaddresses/machine/blob/fa2ea47a7290/docs/install.md

I’d love a review on this from @trescube and @jalessio — I’m a bit stuck on the role of Docker vs. Docker-compose here, and could use some advice on how to think about this. For example, we use Docker-compose to supply PostGIS so that the tests all pass, but it’s not obvious to me how to also use it to support a volume-mapping workflow so that local changes can be run and checked.

[migurski]

Based on advice, I rewrote the local development instructions to use docker-compose instead of plain Docker:

• https://github.com/openaddresses/machine/blob/c08a550/docs/install.md

[migurski]

This seems to work well in EC2 instance user data:

#!/bin/sh
apt-get update -y
apt-get install -y docker.io
docker pull openaddr/machine

[migurski]

This is working well for userdata:

#!/bin/sh -ex
apt-get update -y
apt-get install -y docker.io awscli

docker pull openaddr/machine
aws s3 cp s3://data.openaddresses.io/… /tmp/environment
docker run --env-file /tmp/environment openaddr/machine openaddr-ci-worker -b data.openaddresses.io -v &
docker run --env-file /tmp/environment openaddr/machine openaddr-ci-worker -b data.openaddresses.io -v &
wait

Possible changes:

• Map /tmp volume to let the larger space be used
• Build an AMI with docker.io and awscli installed

[migurski]

I am likely to go live with this as version 5.0.0 of Machine, initially applying it to Worker instances only. I’m planning a crossover period, and then getting rid of the Chef stuff at the end of the crossover at 6.0.0.

[nvkelso]

Congrats!

…

On Apr 19, 2017, at 18:03, Michal Migurski ***@***.***> wrote: I am likely to go live with this as version 5.0.0 of Machine, initially applying it to Worker instances only. I’m planning a crossover period, and then getting rid of the Chef stuff at the end of the crossover at 6.0.0. ― You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.