Restarting socket unit fails with `Address already in use` because `aardvark-dns` inherited the socket

[eriksjolund]

Issue Description

Restarting socket unit fails with Address already in use because aardvark-dns inherited the socket.

Steps to reproduce the issue

1. Create user test

   sudo useradd test
   

2. Start a login shell for the user test

   sudo machinectl shell --uid=test
   

3. Create the file ~/test.bash containing

   #!/bin/bash
   set -o errexit
   mkdir -p  ~/.config/systemd/user
   mkdir -p  ~/.config/containers/systemd
   
   cat << EOF > ~/.config/systemd/user/test.socket
   [Socket]
   ListenStream=8989
   EOF
   
   cat << EOF > ~/.config/containers/systemd/test.network
   [Network]
   EOF
   
   cat << EOF > ~/.config/containers/systemd/test.container
   [Container]
   Image=docker.io/library/alpine
   Exec=sh -c "sleep inf"
   Network=test.network
   EOF
   
   cp ~/.config/containers/systemd/test.container \
      ~/.config/containers/systemd/extra.container
   
   systemctl --user daemon-reload
   systemctl --user start test.socket
   systemctl --user start test.service
   
   ss -H -tlnp '( sport = :8989 )' | awk '{print $NF}' > ~/output.ss.1
   systemctl --user start extra.service
   systemctl --user stop test.service
   systemctl --user stop test.socket
   
   ss -H -tlnp '( sport = :8989 )' | awk '{print $NF}' > ~/output.ss.2
   
   echo =========== create socket again
   systemctl --user --verbose start test.socket
   

4. Run the script

   bash ~/test.bash
   

   The following output is printed

   Stopping 'test.service', but its triggering units are still active:
   test.socket
   =========== create socket again
   Job failed. See "journalctl -xe" for details.
   Jan 03 10:34:23 localhost.localdomain systemd[85269]: test.socket: Failed to create listening socket ([::]:8989): Address already in use
   Jan 03 10:34:23 localhost.localdomain systemd[85269]: test.socket: Failed to listen on sockets: Address already in use
   Jan 03 10:34:23 localhost.localdomain systemd[85269]: test.socket: Failed with result 'resources'.
   Jan 03 10:34:23 localhost.localdomain systemd[85269]: Failed to listen on test.socket.
   

Describe the results you received

The socket unit test.socket could not be recreated. I see the error message Address already in use

In the file output.ss.2 I see that aardvark-dns still has the old socket. That file was produced just before the last step:

systemctl --user --verbose start test.socket

For details:

test@localhost:~$ cat output.ss.1
users:(("sleep",pid=85432,fd=3),("conmon",pid=85430,fd=3),("aardvark-dns",pid=85426,fd=3))
test@localhost:~$ cat output.ss.2
users:(("aardvark-dns",pid=85426,fd=3))
test@localhost:~$ 

Describe the results you expected

Success

podman info output

host:
  arch: arm64
  buildahVersion: 1.42.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-2.fc43.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 99.69
    systemPercent: 0.16
    userPercent: 0.15
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "44"
  emulatedArchitectures:
  - linux/386
  - linux/amd64
  eventLogger: journald
  freeLocks: 2047
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1018
      size: 1
    - container_id: 1
      host_id: 1703936
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1018
      size: 1
    - container_id: 1
      host_id: 1703936
      size: 65536
  kernel: 6.18.0-65.fc44.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 401326080
  memTotal: 4079804416
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.17.0-1.fc44.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.17.0
    package: netavark-1.17.1-1.fc44.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.17.1
  ociRuntime:
    name: crun
    package: crun-1.25.1-1.fc44.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.25.1
      commit: 156ae065d4a322d149c7307034f98d9637aa92a2
      rundir: /run/user/1018/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20251215.gb40f5cd-1.fc44.aarch64
    version: |
      pasta 0^20251215.gb40f5cd-1.fc44.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1018/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-3.fc43.aarch64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.6.0
  swapFree: 0
  swapTotal: 0
  uptime: 34h 47m 44.00s (Approximately 1.42 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/test18/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/test18/.local/share/containers/storage
  graphRootAllocated: 10132369408
  graphRootUsed: 9082601472
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1018/containers
  transientStore: false
  volumePath: /var/home/test18/.local/share/containers/storage/volumes
version:
  APIVersion: 5.7.1
  BuildOrigin: Fedora Project
  Built: 1765324800
  BuiltTime: Wed Dec 10 00:00:00 2025
  GitCommit: f845d14e941889ba4c071f35233d09b29d363c75
  GoVersion: go1.25.5 X:nodwarf5
  Os: linux
  OsArch: linux/arm64
  Version: 5.7.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Fedora CoreOS 44.20251219.91.0 (rawhide)

$ rpm-ostree  status
State: idle
warning: Failed to query journal: couldn't find current boot in journal
Deployments:
● ostree-image-signed:docker://quay.io/fedora/fedora-coreos:rawhide
                   Digest: sha256:32e07fd28e5fa8a9417334ed11b227670c95321eaf8194e2451410438e424502
                  Version: 44.20251219.91.0 (2025-12-19T07:37:08Z)
          LayeredPackages: docker-buildkit docker-buildx emacs-nox golang make

  ostree-image-signed:docker://quay.io/fedora/fedora-coreos:rawhide
                   Digest: sha256:32e07fd28e5fa8a9417334ed11b227670c95321eaf8194e2451410438e424502
                  Version: 44.20251219.91.0 (2025-12-19T07:37:08Z)

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

That is the problem of the normal fork exec inheriting all the fds. We don't have much control over it on the podman side since we cannot just close all fds as the go runtime has a ton of internal ones that we cannot know about.

The best option I see is that aardvark-dns calls close_range() on startup to close all fds > 2 since it should not need any