Provide a way to run the base-image as non-root

[hoshposh]

Overview

There are numerous guidances about running within a container as non-root. These involve the following approaches that layer security around the running container.

1. Use the Dockerfile USER directive to specify the default, preferably non-root user that will run the combination of the ENTRYPOINT and CMD commands.
2. Support the use of the docker run --user=xxx --group=yyy parameters that control which user the container runs as.

This issue is to request either guidance or an approach so that downstream derivatives of the base image can run their commands as non-root.

Notes

1. During our investigation the initial hurdle to being able to use the USER directive within our derived Dockerfile were the /startup/startup.d scripts that install CA certificates.

2. There are no existing utilities installed like gosu that provide a way to perform a privilege step-down from root to a non-root user. This might be a way to execute the /startup scripts as root and then run everything else as a non-root user.

   • Derived images could add this utility, but is this the approach that should be followed?

3. Is an alternative approach to derive a base image after a run of the /startup scripts, so that the image includes the CA certificate changes?

Reference links

• https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b
• https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf
• https://docs.docker.com/engine/security/security/

[dermot-hardy]

SCMOD-11949 raised in Jira so that this issue can be taken into a sprint.

[rorytorneymf]

I've been looking into how we can use the gosu utility to achieve this. I've described some options below.

Option 1: Document how derived containers can add a non-root user. This makes the dervied container responsible for the logic to add the gosu tool and add the non-root user. No code changes here, just a documentation task.

In a derived container, update the Dockerfile to install the gosu utility, and then override the ENTRYPOINT of the base container with a new ENTRYPOINT command in the derived container:

Dockerfile:

# Install gosu
RUN    gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
    && curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64" \
    && curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64.asc" \
    && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
    && rm /usr/local/bin/gosu.asc \
    && chmod +x /usr/local/bin/gosu

# Install startup scripts as root, then switch to non-root user
ADD ./gosu-entrypoint.sh /usr/local/bin/gosu-entrypoint.sh
RUN chmod +x /usr/local/bin/gosu-entrypoint.sh
ENTRYPOINT ["/usr/local/bin/gosu-entrypoint.sh"]
...
# Execute CMD as the non-root user added in gosu-entrypoint.sh
CMD ["whoami"]

The new ENTRYPOINT command in the derived container points to a script that will:

• Call the original ENTRYPOINT from the base container (which will install certs etc as root)
• Create a new, non-root user
• Execute the supplied command as a non-root user

gosu-entrypoint.sh

#!/bin/bash

set -e

# Call original entrypoint
/tini -s /startup/startup.sh

# Add new (non-root) user
useradd --shell /bin/bash --system --user-group --create-home appuser

# Run CMD in Dockerfile as non-root user
exec /usr/local/bin/gosu appuser "$@"

Option 2: Update opensuse-base-image to add the gosu utility and add a new user based on a new GOSU_USER env var. Derived containers just need to set an ENV var to add a new user

In the opensuse-base-image, add the gosu utility:

Dockerfile

 # Install gosu
 RUN gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
 && curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64" \
 && curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64.asc" \
 && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
 && rm /usr/local/bin/gosu.asc \
 && chmod +x /usr/local/bin/gosu

and add the following to the end of the startup.sh file:

startup.sh

# If GOSU_USER environment variable is set, add the user and execute the specified command as that user
if [ -n "$GOSU_USER" ]; then
    log "Adding new user named ${GOSU_USER}..."
    useradd --shell /bin/bash --system --user-group --create-home ${GOSU_USER}
    log "New user named ${GOSU_USER} added. Subsequent commands will be run as this user."

    exec /usr/local/bin/gosu $GOSU_USER "$@"
else
     log "The GOSU_USER environment variable is not set, subsequent commands will be run as the default (root) user."
    # GOSU_USER environment variable is not set, execute the specified command as the default (root) user
    exec "$@"
fi

Now, if a derived container wants to add a non-root user, it just needs to set the GOSU_USER ENV var in its dockerfile:

Dockerfile

ARG DOCKER_HUB_PRIVATE=docker.io
FROM ${DOCKER_HUB_PRIVATE}/dev/cafapi/opensuse-base
ENV GOSU_USER=rory
CMD ["whoami"] # Outputs rory

Otherwise, if it just wants to run as the root user, it does not have to do anything:

Dockerfile

ARG DOCKER_HUB_PRIVATE=docker.io
FROM ${DOCKER_HUB_PRIVATE}/dev/cafapi/opensuse-base
CMD ["whoami"] # Outputs root

The base container will pick up this ENV var if present, add the user, and execute the specified command as that user.

Option 3: Update opensuse-base-image to add the gosu utility and perform command as the user based on a new GOSU_USER env var. Derived containers responsible for (optionally) adding the user and setting GOSU_USER env var

This has the advantage of the base container doing most of the work, but allowing the derived image to perform the 'useradd ...' command, so the
derived container has fine-grained control over the options supplied to 'useradd ...".

Also, in this approach, the derived container can choose to use an existing user, rather than adding a new user with 'useradd ...'

In this approach the base container is still responsible for adding the gosu utility:

Dockerfile

# Install gosu
RUN    gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
    && curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64" \
    && curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64.asc" \
    && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
    && rm /usr/local/bin/gosu.asc \
    && chmod +x /usr/local/bin/gosu

however, it does not add the user, it just executes the command as the user supplied by a derived container:

startup.sh

# If GOSU_USER environment variable is set, execute the specified command as that user
if [ -n "$GOSU_USER" ]; then
    log "The GOSU_USER environment variable has been supplied with a user named ${GOSU_USER}. Subsequent commands will be run as this user.
    # Note no 'useradd' command here
    exec /usr/local/bin/gosu $GOSU_USER "$@"
else
     log "The GOSU_USER environment variable is not set, subsequent commands will be run as the default (root) user."
    # GOSU_USER environment variable is not set, execute the specified command as the default (root) user
    exec "$@"
fi

Now, if a derived container wants to add a non-root user, it needs to RUN the 'useradd' command and set the following ENV var in its dockerfile:

Dockerfile

ARG DOCKER_HUB_PRIVATE=docker.io
FROM ${DOCKER_HUB_PRIVATE}/dev/cafapi/opensuse-base
RUN useradd --system --user-group --create-home rory
ENV GOSU_USER=rory
CMD ["whoami"] # Outputs rory

Otherwise, if it just wants to use an existing user (user must exist in image already), skip the 'useradd' command and just set the ENV var:

Dockerfile

ARG DOCKER_HUB_PRIVATE=docker.io
FROM ${DOCKER_HUB_PRIVATE}/dev/cafapi/opensuse-base
ENV GOSU_USER=rory # Should be already present in image
CMD ["rory"] # Outputs root

And if it wants to use the default root user, it does nothing:

Dockerfile

ARG DOCKER_HUB_PRIVATE=docker.io
FROM ${DOCKER_HUB_PRIVATE}/dev/cafapi/opensuse-base
CMD ["whoami"] # Outputs root

Option 4: Hybrid of option 2 + option 3

This is similar to option 2, though perhaps some more customization could be added for special cases (i.e. you could potentially get the advantages of both 2 and 3 by combining them - checking for the presence of an optional useradd shell script and using it if it has been made available, but falling back to a standard useradd command if it hasn't been).

[dermot-hardy]

• Is GOSU_USER a standard convention? It feels like it is exposing an implementation detail (i.e. that gosu is being used for the permissions step-down). Would something like RUNAS_USER be better?
• If you adjusted option 2 so that you only create the non-privileged user if they do not already exist, then would that effectively give you the advantages of 3/4, because the derived container could customize the creation of the user using a standard startup script, which is of course already supported?

[hoshposh]

@rorytorneymf Thanks for the amount of time you appear to have devoted to the issue, and for providing such a thorough set of options.

• I agree with @dermot-hardy that a RUNAS_USER variable might be a better name conveying the intent rather than the implementation.
• How does this work when you run with the docker run --user=xxx --group=yyy parameters, or does that aspect need to be discussed in a separate issue or thread?

[munifrog]

I like option 2.
Why not have another variable for adduser options, as that is a one-line command anyway?
Something like:

RUNAS_USER_OPTIONS = ${RUNAS_USER_OPTIONS:- --shell /bin/bash --system --user-group --create-home}
...
# If RUNAS_USER environment variable is set, add the user and execute the specified command as that user
if [ -n "$RUNAS_USER" ]; then
    log "Adding new user named ${RUNAS_USER}..."
    if [ -v RUNAS_USER_OPTIONS ]; then
        useradd ${RUNAS_USER_OPTIONS}
    else
        useradd --shell /bin/bash --system --user-group --create-home ${RUNAS_USER}
    fi
    log "New user named ${RUNAS_USER} added. Subsequent commands will be run as this user."

    exec /usr/local/bin/gosu $RUNAS_USER "$@"
else
     log "The RUNAS_USER environment variable is not set, subsequent commands will be run as the default (root) user."
    # RUNAS_USER environment variable is not set, execute the specified command as the default (root) user
    exec "$@"
fi

[rorytorneymf]

How does this work when you run with the docker run --user=xxx --group=yyy parameters, or does that aspect need to be discussed in a separate issue or thread?

I'm not sure how well the above approaches will work with the docker --user-xxx --group=yyy options. I have seen examples where instead of those flags, env vars are used (examples below), not sure if this would be acceptable or not?

Example 1
https://gist.github.com/alexpearce/b438bc9f358ba7b333f2e15e6bd826f0

#!/bin/bash
# Run commands in the Docker container with a particular UID and GID.
# The idea is to run the container like
#   docker run -i \
#     -v `pwd`:/work \
#     -e LOCAL_USER_ID=`id -u $USER` \
#     -e LOCAL_GROUP_ID=`id -g $USER` \
#     image-name bash
# where the -e flags pass the env vars to the container, which are read by this script.
# By setting copying this script to the container and setting it to the
# ENTRYPOINT, and subsequent commands run in the container will run as the user
# who ran `docker` on the host, and so any output files will have the correct
# permissions.

USER_ID=${LOCAL_USER_ID:-9001}
GROUP_ID=${LOCAL_GROUP_ID:-$USER_ID}

echo "Starting with UID : $USER_ID, GID: $GROUP_ID"
groupadd -g $GROUP_ID thegroup
useradd --shell /bin/bash -u $USER_ID -g thegroup -o -c "" -m user
export HOME=/home/user

exec /usr/local/bin/gosu user:thegroup "$@"

Example 2
https://github.com/MiKTeX/docker-miktex-test-opensuse

docker run --rm -t \
  -v ~/work/miktex/test-suite:/miktex/test-suite:ro \
  -v ~/work/miktex/tests/opensuse-latest:/miktex/test:rw \
  -e USER_ID=`id -u` \
  -e GROUP_ID=`id -g` \
  miktex/miktex-test-opensuse:latest

#!/bin/sh -e

if [ -d /miktex/build ]; then
    zypper install -y /miktex/build/*.rpm
else
    echo TODO
fi

GROUP_ID=${GROUP_ID:-1001}
USER_ID=${USER_ID:-1001}

groupadd -g $GROUP_ID -o joe
useradd --shell /bin/bash -u $USER_ID -g $GROUP_ID -o -c "" -m joe
export HOME=/home/joe
exec gosu joe "$@"

[hoshposh]

I'm not sure how well the above approaches will work with the docker --user-xxx --group=yyy options. I have seen examples where instead of those flags, env vars are used (examples below), not sure if this would be acceptable or not?

The main reason for reviewing those options was based on some brief comments from the ITOM team where they mentioned that their customers had requirements for the Pods and ultimately containers to be able to run as certain users in the customer's Kubernetes clusters. So I had assumed that was ultimately achieved through the use of the --user and --group equivalent parameters.

Having said that I do like the two examples of exposing the environment variables for USER_ID and GROUP_ID. So, depending on the insight from others (to clear up my Kubernetes user fuzziness), your approach in the gist examples might be something to be considered.

[dermot-hardy]

Hi Lyndon,

When you say --user=xxx --group=yyy I assume you mean --user=xxx:yyy because there is no --group argument. You seem to be under the impression that that option can be used to run as an arbitrary user that has been setup on the host system, but it is actually just an override for the USER Dockerfile command - i.e. the username specified has to be a user that already exists inside the image.

That said you could use --user=$(id -u xxx):$(id -u yyy) to pass the ids when starting the container, and that would make it run as that user from the host perspective (assuming the same user namespace is being used, which is normally the case). Of course the same goes for using the RUNAS_USER environment variable where you could use the exact same syntax -e RUNAS_USER=$(id -u xxx):$(id -u yyy) to do the same.

The more I think about the more I'm leaning towards option 3, where you just keep it simple and use gosu if RUNAS_USER is specified and don't if it isn't.

It covers the main case where we want the service to be run as a non non-privileged user by default, by adding the necessary useradd (and possibly chown and other commands`) to setup the service user/group and add the environment variable to make this the default user, as Rory outlined in his original comments.

But it also doesn't exclude the possibility of running as a host user if desired (by passing the ids as just discussed).

And if a service has more complex requirements then they can also be easily accommodated by adding an additional startup script.

That's where my thinking is on this at the minute anyway.

Thanks,
Dermot

[hoshposh]

Yes, my snippets should have been using --user=xxx:yyy invocation parameter, sorry for the misrepresented and confusing examples due to my poor notes. Likely due to the Kubernetes securityContext declarations where user (runAsUser) and group (runAsGroup) are separate parameters.

Which leads me to my areas of uncertainty . . .
1. File ownership actions.
2. Work within a Kubernetes environment.

File ownership

How should external volumes be managed so that the container's RUNAS_USER can work with those volumes? Would that be a situation to use the RUNAS_USER=$(id -u):$(id -g), and rely on the person running the container to ensure that identities used have file permissions to the mounted volumes?

When using the default RUNAS_USER would the guidance for derived image maintainers be to create startup scripts to perform chown type actions on mounted volumes?

Kubernetes securityContext

Our ultimate goal is to be able to use derived images in Kubernetes clusters, possibly where runAsUser, runAsGroup are declared in a pod or containers securityContext. My assumption would be that those parameters are turned into the containerd equivalent of docker run --user=xxx:yyy, and if so will the non-root approach work?

[hoshposh]

I should add that my intent is not to throw up hurdles, but in my brief investigations prior to opening this issue, these were some of the hurdles we ran into. So, I am hoping that we can receive guidance on the best approach and more importantly recognize the situations we will not be able to support.

Cheers!

[rorytorneymf]

Our ultimate goal is to be able to use derived images in Kubernetes clusters, possibly where runAsUser, runAsGroup are declared in a pod or containers securityContext. My assumption would be that those parameters are turned into the containerd equivalent of docker run --user=xxx:yyy, and if so will the non-root approach work?

I haven't been able to find a whole lot of information or examples regarding using the above approach (gosu) with the Kubernetes security context, so not sure if it's possible. I've posted a question to try and get some more info:

https://stackoverflow.com/questions/65995431/using-gosu-with-kubernetes-runasuser-security-context

One comment states:

To use gosu, you'ld need securityContext.privileged: true, securityContext.runAsUser: 0, and maybe a PodSecurityPolicy + RBAC granting your ServiceAccount with permissions starting privileged containers. Usually better to go with stuff like nsswrapper, if passwd/group entries are actually required.

[munifrog]

In a meeting I listened to, it was proposed that an init container perform the tasks that require elevate privileges. Then the actual container would be run as non-root. That approach is that same pattern as proposed here with the separation of duties occurring outside the containers and this within them.
And with the tug-of-war between convenience and security, I like the idea of the base container allowing for both.

[dermot-hardy]

Hi Rory,

As you know the consensus from the Container Workgroup discussion last week was that we should endeavour to run as a non-root user from the outset. Customers expect to be able to use the --user=xxx:yyy syntax (or equivalent) to run as an arbitrary user on the host system.

I think we need to take a step-by-step approach to get there. Firstly, let's go ahead and implement your option 3, i.e. making the gosu utility available in the base image and using it automatically if the RUNAS_USER environment variable is set. This will cater for the scenario discussed where an image has a startup script that for some reason must be run as root, but where the main service can be run as a lesser-privileged user. Following on from that let's look at the startup scripts that we include by default, and look specifically at whether it is possible to make adjustments to them to allow them to be run as arbitrary users. Our ultimate aim here must be to facilitate services to run as a non-root user from the outset.

Thanks,
Dermot

[munifrog]

In case it helps, I took the ideas presented here and made a container that runs as an arbitrary user:
Here is my Dockerfile:

ARG BASE_IMAGE=cafapi/opensuse-tomcat:5.2.0
FROM ${BASE_IMAGE} AS build_stage
ENV GOSU_USER=${GUSU_USER:-myself}
COPY match.sh /startup/startup.d/
RUN echo "Install gosu" \
 && gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
 && curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64" \
 && curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64.asc" \
 && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
 && rm /usr/local/bin/gosu.asc \
 && echo "Install helpful packages" \
 && chmod +x /usr/local/bin/gosu \
 && zypper -nq refresh \
 && zypper -nq install --no-recommends \
    vim \
 && zypper clean --all \
 && echo "Define aliases" \
 && echo -e "alias la='ls -la'" > ~/.bashrc \
 && echo "Update startup script for GOSU_USER" \
 && sed -i "s|exec \"\$@\"|if [ -n \"\$GOSU_USER\" ]; then\n  exec /usr/local/bin/gosu \$GOSU_USER \"\$@\"\nelse\n  exec \"\$@\"\nfi|g" "/startup/startup.sh" \
 && echo "Create GOSU_USER" \
 && useradd --system --user-group --create-home "$GOSU_USER" \
 && echo "Enable scripts" \
 && chmod +x /startup/startup.d/match.sh

VOLUME [ "/vol" ]
ENTRYPOINT [ "/bin/bash" ]

... and here is the match.sh script used to change GOSU_USER's UID and GID to match an arbitrary user:

#!/bin/bash

if [ -n "${GOSU_USER}" ]; then
  OWN_LOCATIONS_GROUP=(
    "/home/${GOSU_USER}"
  )
  OWN_LOCATIONS_USER=(
  )

  # Order matters here: GID before UID
  # Root (0) does not need special privileges

  GOSU_USER_GID="$( id -g ${GOSU_USER})"
  if [ "${GOSU_USER_GID}" != "0" ]; then
    VOLUME_GID="$( stat -c '%g' /vol/ )"

    if [ "${GOSU_USER_GID}" != "${VOLUME_GID}" ]; then
      echo "Changing \"${GOSU_USER}\" GID from \"${GOSU_USER_GID}\" to \"${VOLUME_GID}\""
      /usr/sbin/groupmod --gid ${VOLUME_GID} ${GOSU_USER}
    fi
    for file in ${OWN_LOCATIONS_GROUP[@]}; do
      if [ -f "${file}" ]; then
        echo "Changing GID of \"${file}\" to \"${VOLUME_GID}\""
        /usr/bin/chgrp "${VOLUME_GID}" "${file}"
      fi
    done
  fi

  GOSU_USER_UID="$( id -u ${GOSU_USER})"
  if [ "${GOSU_USER_UID}" != "0" ]; then
    VOLUME_UID="$( stat -c '%u' /vol/ )"

    if [ "${GOSU_USER_UID}" != "${VOLUME_UID}" ]; then
      echo "Changing \"${GOSU_USER}\" UID from \"${GOSU_USER_UID}\" to \"${VOLUME_UID}\""
      /usr/sbin/usermod --uid ${VOLUME_UID} ${GOSU_USER}
    fi
    for file in ${OWN_LOCATIONS_USER[@]}; do
      if [ -f "${file}" ]; then
        echo "Changing UID of \"${file}\" to \"${VOLUME_UID}\""
        /usr/bin/chown "${VOLUME_UID}" "${file}"
      fi
    done
  fi
fi

By the time gosu is run, the GOSU_USER has the same UID and GID of the mounted volume. But of course, you could use environment variables or other means to communicate which UID and GID to use.