Watchtower update created hundreds of container instances

[ginkel]

Describe the bug

Hi there,

with yesterday's update to 2.3.1 something went horribly wrong: On all of my servers I see > 1.000 watchtower container instances in the "Created" state and one instance that is spinning and immediately exits again when trying to start up:

time="2025-03-29T08:39:15Z" level=info msg="Found multiple running watchtower instances. Cleaning up."
time="2025-03-29T08:39:15Z" level=info msg="Stopping /watchtower (44c86fd4618a) with SIGTERM"

$ docker ps -a | grep watchtower | wc -l 
1175

$ docker ps -a | grep watchtower

fcdcfd7f3775   beatkind/watchtower:latest                                       "/watchtower --http-…"   10 hours ago   Created                                                                                                                                                                                                                                                   HAJsXGyIpARBfXjfVFJAiGdkSVAXzWZQ

[...]

265b6e09f9ea   beatkind/watchtower:latest                                       "/watchtower --http-…"   20 hours ago   Created                                                                                                                                                                                                                                             XncGLIAlWkBpKygXvXqDLHPBddGluvdh
64a301b58c22   beatkind/watchtower:latest                                       "/watchtower --http-…"   20 hours ago   Created                                                                                                                                                                                                                                             laiRTvQkJsVVTlZVcmWOibRzHkvkElXj
c7991b109e2a   beatkind/watchtower:latest                                       "/watchtower --http-…"   20 hours ago   Created                                                                                                                                                                                                                                             mvKBnpzZdqiGKXxJvcBwGxdOoFYOLPwk
44c86fd4618a   beatkind/watchtower:latest                                       "/watchtower --http-…"   20 hours ago   Exited (2) 4 seconds ago   

This happened on all of my servers running Ubuntu 24.04 or 22.04 and Docker 27.2.1.

Any ideas?

Thanks,
Thilo

Steps to reproduce

1. Wait for Watchtower to perform an auto-update

Expected behavior

Watchtower updates itself without leaving hundreds of broken container behind ;-)

Screenshots

No response

Environment

• Ubuntu 24.04 / 22.04
• x86_64 and arm64
• Docker 27.2.1

Your logs

See above

Additional context

No response

[beatkind]

Oh wow, @ginkel thanks for the report. This shouldn't have happened. Are you able to reproduce this behavior?

[ginkel]

I didn't try so far, but the issue happened on seven servers with a 100% failure rate...

[beatkind]

I tried to reproduce it via the latest-dev tag, but was so far not able to-do so. I am kinda a bit in the dark here what went wrong.

[Marcus1Pierce]

@ginkel Just curious, how do you run Watchtower? For example, what would the docker run command look like, or what would the content of the docker-compose.yaml file look like?

[m4tze]

I had this issue on a server with the old (>1 yr) original containrrr/watchtower image tonight. Maybe that is caused by a docker problem. I had around 600 processes like

runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/cf86e79c5ab130be91dc082b73ff228cdc91c51a097287c8ba325b5d729f0b81/log.json --log-format json --systemd-cgroup exec --process /tmp/runc-process3756386413 --detach --pid-file /run/containerd/io.containerd.runtime.v2.task/moby/cf86e79c5ab130be91dc082b73ff228cdc91c51a097287c8ba325b5d729f0b81/[234ae99fb93e54062ea2a9a0f9fc5200271118e0cf34870469324453d1742bd7.pid](http://234ae99fb93e54062ea2a9a0f9fc5200271118e0cf34870469324453d1742bd7.pid/) cf86e79c5ab130be91dc082b73ff228cdc91c51a097287c8ba325b5d729f0b81

Found this issue when searching an alternative for the containrrr image. I will update to the latest beatkind image now, thanks for updating it @beatkind!

[Minituff]

I just ran into this issue as well on Version 2.3.2