Fixed MAC address not preserved during update

[mammo0]

Describe the bug

Hello,

this bug is also present in the upstream repository: containrrr#1941

In short: If you set a fixed MAC address in a container, it gets replaced by a generated one after an update with watchtower.

Steps to reproduce

1. Create a container and set a fixed MAC address.
2. Let watchtower update the container (works without errors)
3. Check the MAC address of the updated container

Expected behavior

The updated container has the fixed MAC address.

Screenshots

No response

Environment

• QNAP NAS QTS 5.2.3 (20250108)
• Kernel: 5.10.60-qnap
• Architecture: x86_64
• Docker Version: 27.1.2-qnap4 (API: 1.46)

Your logs

no error is thrown

Additional context

No response

[beatkind]

@mammo0 which Version of watchtower are you running?

[mammo0]

Sry, the latest release 2.2.0

[beatkind]

Can you try running with the follwoing command / ENV var: https://watchtower.devcdn.net/arguments/#docker_api_version

And that set to "1.46"?

[mammo0]

I will try it. I've set the value and I think it will only take a couple of days until an update is available for one of my containers. Then I will report back.

[mammo0]

Unfortunately setting DOCKER_API_VERSION=1.46 did not work.

There was an update for the mariadb image. I have also a nextcloud container that depends on mariadb (I set the dependency with com.centurylinklabs.watchtower.depends-on).
During the update process both containers were recreated. Afterwards both had the default MAC address that gets set from docker and not the custom one they had before.

Here are the debug logs from watchtower:

time="2025-03-25T13:59:45+01:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2025-03-25T13:59:46+01:00" level=debug msg="Making sure everything is sane before starting"
time="2025-03-25T13:59:46+01:00" level=info msg="Watchtower 2.2.0"
time="2025-03-25T13:59:46+01:00" level=info msg="Using no notifications"
time="2025-03-25T13:59:46+01:00" level=info msg="Only checking containers using enable label"
time="2025-03-25T13:59:46+01:00" level=info msg="Running a one time update."
time="2025-03-25T13:59:46+01:00" level=debug msg="Checking containers for updated images"
time="2025-03-25T13:59:46+01:00" level=debug msg="Retrieving running containers"
time="2025-03-25T13:59:47+01:00" level=debug msg="Trying to load authentication credentials." container=/watchtower image="beatkind/watchtower:latest"
time="2025-03-25T13:59:47+01:00" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
time="2025-03-25T13:59:47+01:00" level=debug msg="Got image name: beatkind/watchtower:latest"
time="2025-03-25T13:59:47+01:00" level=debug msg="Checking if pull is needed" container=/watchtower image="beatkind/watchtower:latest"
time="2025-03-25T13:59:47+01:00" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2025-03-25T13:59:47+01:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2025-03-25T13:59:47+01:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2025-03-25T13:59:47+01:00" level=debug msg="Setting scope for auth token" image=docker.io/beatkind/watchtower scope="repository:beatkind/watchtower:pull"
time="2025-03-25T13:59:47+01:00" level=debug msg="No credentials found."
time="2025-03-25T13:59:47+01:00" level=debug msg="Parsing image ref" host=index.docker.io image=beatkind/watchtower normalized=docker.io/beatkind/watchtower tag=latest
time="2025-03-25T13:59:47+01:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/beatkind/watchtower/manifests/latest"
time="2025-03-25T13:59:48+01:00" level=debug msg="Found a remote digest to compare with" remote="sha256:89cb81f37fe07a8bd84df21f8b816d47bcfc99c8730ae98a3afa7fdc4995bfdb"
time="2025-03-25T13:59:48+01:00" level=debug msg=Comparing local="sha256:89cb81f37fe07a8bd84df21f8b816d47bcfc99c8730ae98a3afa7fdc4995bfdb" remote="sha256:89cb81f37fe07a8bd84df21f8b816d47bcfc99c8730ae98a3afa7fdc4995bfdb"
time="2025-03-25T13:59:48+01:00" level=debug msg="Found a match"
time="2025-03-25T13:59:48+01:00" level=debug msg="No pull needed. Skipping image."
time="2025-03-25T13:59:48+01:00" level=debug msg="No new images found for /watchtower"
time="2025-03-25T13:59:55+01:00" level=debug msg="Trying to load authentication credentials." container=/nextcloud image="linuxserver/nextcloud:latest"
time="2025-03-25T13:59:55+01:00" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
time="2025-03-25T13:59:55+01:00" level=debug msg="Got image name: linuxserver/nextcloud:latest"
time="2025-03-25T13:59:55+01:00" level=debug msg="Checking if pull is needed" container=/nextcloud image="linuxserver/nextcloud:latest"
time="2025-03-25T13:59:55+01:00" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2025-03-25T13:59:56+01:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2025-03-25T13:59:56+01:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2025-03-25T13:59:56+01:00" level=debug msg="Setting scope for auth token" image=docker.io/linuxserver/nextcloud scope="repository:linuxserver/nextcloud:pull"
time="2025-03-25T13:59:56+01:00" level=debug msg="No credentials found."
time="2025-03-25T13:59:56+01:00" level=debug msg="Parsing image ref" host=index.docker.io image=linuxserver/nextcloud normalized=docker.io/linuxserver/nextcloud tag=latest
time="2025-03-25T13:59:56+01:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/linuxserver/nextcloud/manifests/latest"
time="2025-03-25T13:59:56+01:00" level=debug msg="Found a remote digest to compare with" remote="sha256:962ffa722d0c1eadb588315baa7ff1009974206588d8d05ac4399e0c6fa95de5"
time="2025-03-25T13:59:56+01:00" level=debug msg=Comparing local="sha256:962ffa722d0c1eadb588315baa7ff1009974206588d8d05ac4399e0c6fa95de5" remote="sha256:962ffa722d0c1eadb588315baa7ff1009974206588d8d05ac4399e0c6fa95de5"
time="2025-03-25T13:59:56+01:00" level=debug msg="Found a match"
time="2025-03-25T13:59:56+01:00" level=debug msg="No pull needed. Skipping image."
time="2025-03-25T13:59:56+01:00" level=debug msg="No new images found for /nextcloud"
time="2025-03-25T14:00:15+01:00" level=debug msg="Trying to load authentication credentials." container=/mariadb image="lscr.io/linuxserver/mariadb:latest"
time="2025-03-25T14:00:15+01:00" level=debug msg="No credentials for lscr.io found" config_file=/config.json
time="2025-03-25T14:00:15+01:00" level=debug msg="Got image name: lscr.io/linuxserver/mariadb:latest"
time="2025-03-25T14:00:15+01:00" level=debug msg="Checking if pull is needed" container=/mariadb image="lscr.io/linuxserver/mariadb:latest"
time="2025-03-25T14:00:15+01:00" level=debug msg="Built challenge URL" URL="https://lscr.io/v2/"
time="2025-03-25T14:00:16+01:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:user/image:pull\"" status="401 Unauthorized"
time="2025-03-25T14:00:16+01:00" level=debug msg="Checking challenge header content" realm="https://ghcr.io/token" service=ghcr.io
time="2025-03-25T14:00:16+01:00" level=debug msg="Setting scope for auth token" image=lscr.io/linuxserver/mariadb scope="repository:linuxserver/mariadb:pull"
time="2025-03-25T14:00:16+01:00" level=debug msg="No credentials found."
time="2025-03-25T14:00:16+01:00" level=debug msg="Parsing image ref" host=lscr.io image=linuxserver/mariadb normalized=lscr.io/linuxserver/mariadb tag=latest
time="2025-03-25T14:00:16+01:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://lscr.io/v2/linuxserver/mariadb/manifests/latest"
time="2025-03-25T14:00:16+01:00" level=debug msg="Could not do a head request for \"lscr.io/linuxserver/mariadb:latest\", falling back to regular pull." container=/mariadb image="lscr.io/linuxserver/mariadb:latest"
time="2025-03-25T14:00:16+01:00" level=debug msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://ghcr.io/token\\\",service=\\\"ghcr.io\\\",scope=\\\"repository:linuxserver/mariadb:pull\\\"\"" container=/mariadb image="lscr.io/linuxserver/mariadb:latest"
time="2025-03-25T14:00:16+01:00" level=debug msg="Pulling image" container=/mariadb image="lscr.io/linuxserver/mariadb:latest"
time="2025-03-25T14:00:41+01:00" level=info msg="Found new lscr.io/linuxserver/mariadb:latest image (fee5e316495c)"
time="2025-03-25T14:00:48+01:00" level=debug msg="container is linked to restarting" linked=/nextcloud restarting=/mariadb
time="2025-03-25T14:00:51+01:00" level=info msg="Stopping /nextcloud (363a3e73b098) with SIGTERM"
time="2025-03-25T14:00:56+01:00" level=debug msg="Removing container 363a3e73b098"
time="2025-03-25T14:00:59+01:00" level=info msg="Stopping /mariadb (d4f4dbaa6c6e) with SIGTERM"
time="2025-03-25T14:01:04+01:00" level=debug msg="Removing container d4f4dbaa6c6e"
time="2025-03-25T14:01:04+01:00" level=debug msg="This is the watchtower container /watchtower"
time="2025-03-25T14:01:04+01:00" level=info msg="Creating /mariadb"
time="2025-03-25T14:01:05+01:00" level=debug msg="Starting container /mariadb (a9cd3bfbb8f9)"
time="2025-03-25T14:01:10+01:00" level=info msg="Creating /nextcloud"
time="2025-03-25T14:01:11+01:00" level=debug msg="Starting container /nextcloud (932b600906ae)"
time="2025-03-25T14:01:18+01:00" level=info msg="Removing image e081f0ba2ec0"
time="2025-03-25T14:01:19+01:00" level=debug msg="Image removal completed" deleted="e081f0ba2ec0, e2491f6c2bfc, 535f7eb43a3b" untagged="lscr.io/linuxserver/mariadb@sha256:31ccc6a761a8"
time="2025-03-25T14:01:20+01:00" level=info msg="Session done" Failed=0 Scanned=2 Updated=1 notify=no
time="2025-03-25T14:01:20+01:00" level=info msg="Waiting for the notification goroutine to finish" notify=no

[beatkind]

@mammo0 I see where the issue is. @nicholas-fedor already fixed it in his fork, I will try to implement a fix also in here.

[nicholas-fedor]

This is an issue that has certainly come up in since Docker updated it's handling of MAC addresses, as described in this post: moby/moby#47441 (comment)

The fix is described here: moby/moby#47474 (comment)

The heart of my attempted fix can be found here: https://github.com/nicholas-fedor/watchtower/blob/main/pkg/container/container_source.go#L374

After many rounds of testing, I found that it was necessary to be as intentional as possible about persisting the container's networking configuration in order to reliably migrate it to the new container.

As the fix is not compatible with clients using API versions less than 1.44, I ultimately opted to use a functional "legacy" version of the fix that doesn't necessarily explicitly handle MAC addressing; however, I have seen the addressing persist in testing.

tl;dr - Try to use a Docker host running API version > 1.44, if possible, but also the legacy Watchtower code for this particular portion of handling containers is definitely outdated and should be fixed. My method is just one way of doing it.

[mammo0]

I switched to @nicholas-fedor's fork and can confirm that the MAC addresses get preserved after container updates.

[andreasruxanda]

Just here to also confirm that @nicholas-fedor's fork works great and it fixes the MAC addreess issue.