Docker image build process

[meeDamian]

As there's absolutely no way to have deterministic builds of Docker images 😭, we need to figure out the best way forward.

Our images need to be build in 2-3 architectures: amd64, arm32v6 and/or aarch64. The first one is absolutely trivial to build on either (Travis, Docker Cloud, Codeship, circleci etc…). However, the last two are a bit more challenging, as neither of the popular CI tools offers builds native to these architectures.

AFAIK there are three main paths forward. Let's discuss them.

1. Cross-compile using qemu

• The biggest advantage of this approach is that it offers users the highest degree of auditability - they can look how build/deploy pipeline is setup, see build logs, and verify that the log corresponds to a correct Dockerfile and that the container IDs match between build log and images hosted on Docker Cloud
• The biggest disadvantage is the necessity to use qemu:
  • in most cases (pretty much anything other than bitcoin-core), it's enough to DL a qemu-arm-static binary from an official source, and just COPY it into the correct location in the container and alter FROM sources to explicitly request images in the desired architecture. Example of all pre-build steps here.
  • the case of bitcoin-core is unfortunately much more involved and hacky/patchy, as currently only Codeship has an execution time that's long enough, and AFAIK they don't allow for neither sudo access nor execution of any commands prior to image build. And because of that building bitcoin-core requires all these files (some can be compiled from source directly some others need to be DLed from a Github repo) as well as some extra changes to the Dockerfile itself (example)
  • I think qemu is also not guaranteed to be correct as it's just a mere CPU emulation layer
• Using a big/popular CI provider also somewhat ensures they won't risk their reputation to tamper with the images (esp. as there are probably other images build built on their platform that would be more profitable to tamper with)
• it's possible some of the CI providers will eventually offer servers with different architecture

2. Our own dedicated build server

• The biggest advantage here is that we don't need to deal with qemu at all, and we can script building of native images
• The biggest disadvantage is that it's a black box for anyone using our images - they need to trust us completely as to whether or not the images were tempered with
• and even though our box is "bare metal", it's still someone else's computer so we need to trust them not to temper with our process
• can be scripted easier than # 3
• needs to be paid for periodically

3. Builds on our own HW device

• The biggest advantage and disadvantage are exactly the same as above
• the difference is that we don't need to trust the company hosting the device
• this option is likely also slower than the 2nd one
• likely requires manual builds every time an update to bitcoin core, lnd etc is released
• might be one of the devices we use anyway, so no extra maintenance effort nor cost is involved

Tl;dr:

1st: much easier to audit for anyone; need to use qemu; completely scriptable;
2nd: a lot of trust involved; maybe easier to script; needs maintenance; costs;
3rd: n-1 trust involved; more manual; needs no extra maintenance; no extra cost;

Note: if I did miss something please let me know, so we can get it to be more complete.

[meeDamian]

My personal order of preference is 1 -> 3 -> 2 (as I'll be the -1 in n-1 😉)

[meeDamian]

Ok, it seems that aarch64 is off the table for now… So the only extra supported arch would be arm32v6.

[nolim1t]

I’m more in favor for an actual box rather than cross compiling

[meeDamian]

Do you know any working CI that allows us to avoid that? If no, how do we do builds?

[nolim1t]

Gitlab ci

[AnotherDroog]

What about setting up some open-source CI like Jenkins and using our home boxes or the hosted one to run jobs?

At least this gives us some of the convenience of a CI system and lets us share resources