401 Unauthorized when using private ECR image in FROM

[antonysouthworth-halter]

Describe the bug

I have a Dockerfile that looks like this:

# syntax=docker/dockerfile:1

FROM xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image:my-tag
# ... other stuff; the build doesn't even get this far

The build falls over pretty much immediately:

env AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$AWS_ACCESS_KEY_ID" build 2>&1
INFO[0000] building image
#1 [internal] load build definition from Dockerfile
#1 sha256:8d67fdf14badea11ba1493900a480c02350ddc9e98fe087c42549f52cbd25745
#1 transferring dockerfile: 32B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:8a25834ab0b08df91cd3f5488c9996ba5141bb998e06c3a3958c9031a4af1a12
#2 transferring context: 34B done
#2 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:1
#3 sha256:ac072d521901222eeef550f52282877f196e16b0247844be9ceb1ccc1eac391d
#3 DONE 1.6s

#4 docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021
#4 sha256:94d31f7191390ebf1aa76bc23bdbd269fd7a0432f853889e4918b44513a34946
#4 resolve docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 done
#4 CACHED

#5 [internal] load build definition from Dockerfile
#5 sha256:abd879327137e3a3e7e0b87463aeff911bfbc148f94954012ad18b309a531ed7
#5 DONE 0.0s

#6 [internal] load metadata for xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image:latest
#6 sha256:3ae0be8977208b9d964dcb043c5e9a3a79046fb7a6b4aeed293d2f8f263305e6
#6 ERROR: unexpected status code [manifests latest]: 401 Unauthorized
------
 > [internal] load metadata for xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image:latest:
------
Dockerfile:4
--------------------
   3 |
   4 | >>> FROM xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image:my-tag
   5 |     RUN \
   6 |       export DEBIAN_FRONTEND=noninteractive \
--------------------
error: failed to solve: rpc error: code = Unknown desc = 156399527556.dkr.ecr.ap-southeast-2.amazonaws.com/aws-athena/python-3.9-slim-buster:latest: unexpected status code [manifests latest]: 401 Unauthorized
FATA[0002] failed to build: build: exit status 1
FATA[0002] failed to run task: exit status 1

As you can see, I already ensure the credentials are passed to build so I dunno why I still get 401.

Reproduction steps

Run build with the above Dockerfile.

Expected behavior

It should not fail on FROM statement.

Additional context

No response

[antonysouthworth-halter]

Actually, an extra question expanding on the above, it seems like there is no way to provide credentials to the buildkit daemon at all? E.g. Docker Hub or ECR or other public or private registry.

For some additional context; some of our CI builds were failing due to exhausting the available free rate limit for Docker Hub, so we thought we could just cache the base image we are using in ECR and then update our Dockerfiles to use it, or e.g. set up Pull-Through cache for ECR, but ran into the above issue.

[antonysouthworth-halter]

We were using the python:3.9-slim-buster image which is available in public.ecr.aws so we just switched to that as a workaround.

[pedrodsrodrigues]

We had a similar problem. We needed to pull an image from an AWS private repository through the FROM instruction in a Dockerfile.
The workaround we ended up using was installing the aws-cli and docker, logging in, and then building the image. Below is an example:

Pipeline:

(...)
- task: build
  privileged: true
  config:
    platform: linux
    image_resource:
      type: registry-image
      source:
        repository: concourse/oci-build-task
    params:
      DOCKERFILE: project/Dockerfile
      CONTEXT: project/
      AWS_ACCESS_KEY_ID: ((data.aws_key))
      AWS_SECRET_ACCESS_KEY: ((data.aws_secret))
      AWS_DEFAULT_REGION: ((data.aws_region))
    inputs:
    - name: project
    run:
      path: project/build.sh

project/build.sh:

#!/usr/bin/env sh

apk add aws-cli docker
docker login --username AWS --password "$(aws ecr get-login-password --region $AWS_DEFAULT_REGION)" <repository endpoint>
build

Being able to pull an image from a private repository seems like a great feature to add, since it's a very common use case, just like concourse/registry-image-resource.

[Dakad]

@antonysouthworth-halter @pedrodsrodrigues You can use a docker build argument ARG BASE_IMAGE=xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image and pass it to the oci-build-task params with IMAGE_ARG_BASE_IMAGE and=docker-base/image.tar

The Dockerfile should look like so

ARG BASE_IMAGE=xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image
ARG BASE_IMAGE_TAG=my-tag
FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG}

....

And on the CI pipeline

---
resource:
# Docker images
- name: docker-base
  type: registry-image
  source:
    repository: apptweak-app
    tag: base
    aws_access_key_id: ((AWS_ACCESS_KEY_ID))
    aws_secret_access_key: ((AWS_SECRET_ACCESS_KEY))
    aws_region: ap-southeast-2

...

jobs:
- name: build-app
  plan:
  - in_parallel:
    - get: git-app
      params:
        depth: 1
      trigger: true
    - get: docker-base
      params:
        format: oci
  - task: build-image
    privileged: true
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: concourse/oci-build-task
          username: ((DOCKERHUB_USERNAME))
          password: ((DOCKERHUB_PASSWORD))
      params:
        IMAGE_ARG_BASE_IMAGE: docker-base/image.tar
      inputs:
        - name: git-app
          path: .
        - name: docker-base
      outputs:
        - name: built-image
      run:
        path: build

[Dakad]

But I aggree, being able to pull the private image directy from the oci-build-task run will be more convenient. 😄

[pedrodsrodrigues]

@antonysouthworth-halter @pedrodsrodrigues You can use a docker build argument ARG BASE_IMAGE=xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image and pass it to the oci-build-task params with IMAGE_ARG_BASE_IMAGE and=docker-base/image.tar

The Dockerfile should look like so

ARG BASE_IMAGE=xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image
ARG BASE_IMAGE_TAG=my-tag
FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG}

....

And on the CI pipeline

---
resource:
# Docker images
- name: docker-base
  type: registry-image
  source:
    repository: apptweak-app
    tag: base
    aws_access_key_id: ((AWS_ACCESS_KEY_ID))
    aws_secret_access_key: ((AWS_SECRET_ACCESS_KEY))
    aws_region: ap-southeast-2

...

jobs:
- name: build-app
  plan:
  - in_parallel:
    - get: git-app
      params:
        depth: 1
      trigger: true
    - get: docker-base
      params:
        format: oci
  - task: build-image
    privileged: true
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: concourse/oci-build-task
          username: ((DOCKERHUB_USERNAME))
          password: ((DOCKERHUB_PASSWORD))
      params:
        IMAGE_ARG_BASE_IMAGE: docker-base/image.tar
      inputs:
        - name: git-app
          path: .
        - name: docker-base
      outputs:
        - name: built-image
      run:
        path: build

Great solution, thanks for sharing! Unfortunately we are using multi-arch images, which seems kind of incompatible with IMAGE_ARG: #110

[DanHartman]

I was able to have the oci-build-task successfully pull from a private registry. The workaround I tried was two fold.

• Add a pre-task that generates a valid .docker/config.json file containing the auth required.
• Use the env var DOCKER_CONFIG in the oci-build-task to dictate which directory should be used to look for this auth file(https://docs.docker.com/engine/reference/commandline/cli/#configuration-files)

jobs:
- name: hello-world
  plan:
  - task: login
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: alpine
      outputs:
      - name: docker-config
      run:
        path: /bin/sh
        args:
        - -c
        - |-
          mkdir -p docker-config
          cat <<EOF > docker-config/config.json
          {
            "auths": {
              "<some-registry-server>": {
                "auth": "<some-base64-encoded-string>"
              }
            }
          }
          EOF
  - task: oci-build
    privileged: true
    config:
      inputs:
      - name: docker-config
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: concourse/oci-build-task
      params:
        OUTPUT_OCI: true
        DOCKER_CONFIG: docker-config
      run:
        path: build

[kanngiesser]

As another workaround which worked for us, the registry-image resource can be used to pull a private image from ECR in the OCI format.
A task input is used to pass the tarball to the build job. The parameter IMAGE_ARG_* for the build job allows to use the tarball as base image for building.

[taylorsilva]

A few things:

• You can load multi-arch images now thanks to load OCI images with IMAGE_ARG_* #130
• The registry-image can pull all platforms for multi-arch images when format: oci-layout is used in the params of the get step
• Combining the previous two points, one can now write a pipeline that avoids the auth issue all together

Closing as now all tools make this workflow possible.