Docker Round-Up

[migurski]

As part of my work with Mapbox, I am focusing on making the OpenAddresses codebase easier to install and use for developers. Based on recent conversations and research, I've identified a few use cases:

• Getting Machine up and running for direct development of the code (mostly me)
• Using Machine locally for rapid iteration on new data sources (e.g. @trescube)
• Adapting Machine for new types of data, such as parcels

Some of the difficulties installing and using Machine include:

• Compiled requirements such as GDAL 2.1.0 require too much toolchain understanding
• Chef installation scripts assume ownership of an entire Linux machine
• Big differences between how Machine is used locally vs. on AWS can make docs complex

Over the past couple years, Docker has emerged as a potential approach toward speeding up local installation of Linux software regardless of host OS. This has come up in OA a few times, e.g. #159 and #547. Conversations with a few developers suggest that the presence of a virtual machine configuration such as Vagrant or Docker is a signal that code is easy to approach and painless to get running. As a stopgap that eases local installation without duplicating the work in our Chef recipes, @jalessio has recommended Vagrant in PR #559. It’s a big improvement over the current situation, and retains our existing use of Chef under CI and in production.

Docker may be a good long-term solution. We need to validate against the three main environments where Machine code is run:

1. Locally on a developer’s Mac, Windows, or Linux computer with continuous edits to the code
2. In a CI environment like Travis or Circle CI for ensuring quality of new code changes
3. In production on AWS where we use Cloudwatch metrics, Auto Scaling Groups, and other features to maintain a running service which can respond to spikes in demand from OA contributors

If Docker's going to work for us, I think it needs to work cleanly on each of these three environments, successfully replace Chef, and not introduce substantial new pain for developers or contributors. So far, I’ve done a few things:

• Created this Dockerfile for Machine based on Chef recipes
• Tested build times and found that it’s slower than Chef when starting from scratch, but not unreasonable
• Checked build sizes and found that all of Machine and all its dependencies can be packed into a ~400MB Docker image
• Tested using Docker in EC2 User Data scripts and found that it works just fine

I expect to work on this over the next few weeks, and I would love feedback and input.

[sbma44]

Nice! My experience with Docker suggests that this is going to work out well. Two quick reactions:

• of the environments you've outlined, CI is very likely to be trickiest. We have often fudged this by having Circle remain relatively close to our Docker environment, but that is, uh, not ideal. Some recent progress has been made on this front I believe, but I'm very keen to see what you find out.
• on image sizes: 400MB sounds pretty manageable but if you want to slim down even more, keep Docker's copy-on-write/layer cache in mind. Cramming every apt-get invocation into a single huge RUN is terrible for development but does save some space once you have everything working, especially if you end that RUN with something like:

    apt-get autoremove -y && \
    rm -rf /var/lib/apt/lists/*

(or whatever else you can get away with)

[migurski]

Circle 2.0 is supposed to be somehow Docker-connected, based on conversations with them in the past. We happen to use Travis here, but we've used both interchangeably in the past and could again.

[migurski]

Also, I noticed that Travis tried to infer some Docker magic from the branch. I think your intuition that CI will be the hard part is right, especially as these companies are moving targets trying to integrate containers into their flows.

[NelsonMinar]

I'm glad you're working on this. I wrote a big chunk of machine code and even I struggle to develop on it now. Last time I ended up creating a whole Virtualbox VM just so I could run Chef to install stuff so I could then develop. Our code isn't even that complex; GDAL is the only sort of tricky library there. But the way the development environment assumes something much like the Chef production environment makes development harder.

So I like the idea of making it simpler to get a development environment going. I also agree with the principle that the three environments (dev, CI, prod) should have as much in common as possible, including the package management.

I have no opinion on Vagrant vs Docker vs Chef, except that Chef seems to be awfully confusing every time I use it. Also the Chef setup we have now doesn't have an isolated environment for the development machine (hence my Virtualbox setup.)

400MB seems like an awfully big image. For comparison I'm seeing numbers like 80MB for a small Ubuntu Docker setup with a simple app like nano.

[migurski]

Nelson, thanks for your comments. Something you might find interesting is the newly-landed Vagrant support documented here: https://github.com/openaddresses/machine/blob/master/docs/install.md#local-development; it's a half-measure on the way to a possible Docker future, but I think it addresses some of the stuff you're describing here.

The 400MB size mostly consists of all the GDAL-and-friends libraries that we're adding. The base Ubuntu image compresses down to 50MB.

[jalessio]

FWIW, a 400MB Docker image size doesn't phase me at all. I'm fairly certain it could be made smaller, but just as a point of comparison for the VirtualBox+Vagrant approach you have to first download a 469 MB VM and then run Chef whichs downloads GDAL-and-friends. I don't have firm numbers but the total over-the-wire getting started size must be ~1GB. So, as a starting point, 400MB seems OK.

[sbma44]

one other thing that is maybe worth consideration: keeping a base-image branch that installs heavyweight dependencies and creates an image to serve as the FROM target for the Dockerfile in master. Assuming you stick the resulting image in ECR or another repository, this has a couple of advantages:

• building new images when developing becomes incredibly fast
• the ubuntu PPA's habit of nuking GDAL packages without warning becomes less of an ongoing headache -- the base image will still be around even when the PPA has betrayed you, making upgrading packages a when-it's-convenient rather than a complete blocker.

[migurski]

I like the base image idea. I duplicated the GDAL packages into our own PPA after getting bitten once by the disappearing act, but the installation still takes forever. Are images similar to tags in git, where there's a public name that points to an immutable underlying SHA?

[ingalls]

@migurski Yup you got it! Once you create the base-image you can upload it to Docker or Amazon ECR and then reference it's tag in the child image.

For example we use ECR and at the top of the second (child) image we would use:
FROM <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/<REPO_NAME>:<REPO_TAG>

[iandees]

I'd love for us to try and talk the openaddressesuk folks in to letting us post to their docker hub namespace before we set up a private ECR.

[migurski]

Are Docker Hub and ECR our only options? I have been saving and loading images via S3; is there a reason that's a bad idea?

[iandees]

I'm interested in Docker Hub because it means our stuff is a bit more discoverable. Images in ECR or S3 are a bit harder to find (other than using them via our docs).

[migurski]

That makes sense, thanks!