fatal error: concurrent map writes

[sreedevk]

Hello,
Firstly, thank you for maintaining this project. it has made my life a lot easier.

I've been running a self hosted dockerized instance of selfhst/icons for about a month now and I realized that it crashes and brings the docker container down when the number of concurrent requests exceeds a certain threshold. From my very limited analysis, the threshold I mentioned is around 120 requests concurrent requests. The logs are attached below. I have not placed any resource constraints on the docker container.

I use nix to setup and run the docker containers. here's the nix virtualisation.oci-containers.containers entry for the icons container.

Please let me know if there's any additional information I can provide to help triage this issue.

icons = {
  autoStart = true;
  image = "ghcr.io/selfhst/icons:latest";
  pull = "newer";
  ports = [
    "${opts.ports.icons}:4050"
  ];
  environment = {
    PORT                 = "4050";
    ICON_SOURCE          = "local"; # "remote"
    STANDARD_ICON_FORMAT = "png"; # webp
  };
  volumes = [
    "/etc/icons/png:/app/icons/png:ro"
  ];
  extraOptions = [
    "--add-host=${opts.hostname}:${opts.addresses.lan.${opts.hostname}}"
    "--no-healthcheck"
    "--dns=${opts.addresses.lan.${opts.hostname}}"
  ];
};

Jan 31 07:16:18 xxxxx icons[8862]: fatal error: concurrent map writes
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 658 [running]:
Jan 31 07:16:18 xxxxx icons[8862]: internal/runtime/maps.fatal({0x782511?, 0x0?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/panic.go:1046 +0x18
Jan 31 07:16:18 xxxxx icons[8862]: internal/runtime/maps.(*Map).Delete(0xc0000c2b70, 0x727820, 0xc00059d698)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/runtime/maps/map.go:652 +0x4c
Jan 31 07:16:18 xxxxx icons[8862]: main.(*Cache).Get(0xc0000c2ba0, {0xc0000122b0, 0xd})
Jan 31 07:16:18 xxxxx icons[8862]:         /app/main.go:57 +0x15d
Jan 31 07:16:18 xxxxx icons[8862]: main.handleIcon({0x7ffac8, 0xc0002a60f0}, 0xc00064a500)
Jan 31 07:16:18 xxxxx icons[8862]:         /app/main.go:248 +0x345
Jan 31 07:16:18 xxxxx icons[8862]: net/http.HandlerFunc.ServeHTTP(0xc0000e8000?, {0x7ffac8?, 0xc0002a60f0?}, 0x63e136?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2322 +0x29
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*ServeMux).ServeHTTP(0x473019?, {0x7ffac8, 0xc0002a60f0}, 0xc00064a500)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2861 +0x1c7
Jan 31 07:16:18 xxxxx icons[8862]: net/http.serverHandler.ServeHTTP({0xc0003465c0?}, {0x7ffac8?, 0xc0002a60f0?}, 0x6?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3340 +0x8e
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).serve(0xc0000de510, {0x7fff70, 0xc0000c2ed0})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2109 +0x665
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*Server).Serve in goroutine 1
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3493 +0x485
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 1 [IO wait]:
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.runtime_pollWait(0x7fc97605ac00, 0x72)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/netpoll.go:351 +0x85
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).wait(0xc0000fc000?, 0x900417577?, 0x0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).waitRead(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*FD).Accept(0xc0000fc000)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_unix.go:613 +0x28c
Jan 31 07:16:18 xxxxx icons[8862]: net.(*netFD).accept(0xc0000fc000)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/fd_unix.go:161 +0x29
Jan 31 07:16:18 xxxxx icons[8862]: net.(*TCPListener).accept(0xc0000e21c0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/tcpsock_posix.go:159 +0x1b
Jan 31 07:16:18 xxxxx icons[8862]: net.(*TCPListener).Accept(0xc0000e21c0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/tcpsock.go:380 +0x30
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*Server).Serve(0xc000024100, {0x7ffbb8, 0xc0000e21c0})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3463 +0x30c
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*Server).ListenAndServe(0xc000024100)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3389 +0x72
Jan 31 07:16:18 xxxxx icons[8862]: net/http.ListenAndServe(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3704
Jan 31 07:16:18 xxxxx icons[8862]: main.main()
Jan 31 07:16:18 xxxxx icons[8862]:         /app/main.go:447 +0x471
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 657 [sync.RWMutex.Lock]:
Jan 31 07:16:18 xxxxx icons[8862]: sync.runtime_SemacquireRWMutex(0x0?, 0x0?, 0x4c82d3?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/sema.go:105 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: sync.(*RWMutex).Lock(0xa611a0?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/sync/rwmutex.go:155 +0x6b
Jan 31 07:16:18 xxxxx icons[8862]: main.(*Cache).Set(0xc0000c2ba0, {0xc0002980d8, 0x16}, {0xc0006bc000, 0x502a})
Jan 31 07:16:18 xxxxx icons[8862]:         /app/main.go:65 +0x65
Jan 31 07:16:18 xxxxx icons[8862]: main.handleIcon({0x7ffac8, 0xc00022c1e0}, 0xc00064a3c0)
Jan 31 07:16:18 xxxxx icons[8862]:         /app/main.go:328 +0xf59
Jan 31 07:16:18 xxxxx icons[8862]: net/http.HandlerFunc.ServeHTTP(0xc0000e8000?, {0x7ffac8?, 0xc00022c1e0?}, 0x63e136?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2322 +0x29
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*ServeMux).ServeHTTP(0x473019?, {0x7ffac8, 0xc00022c1e0}, 0xc00064a3c0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2861 +0x1c7
Jan 31 07:16:18 xxxxx icons[8862]: net/http.serverHandler.ServeHTTP({0xc000346540?}, {0x7ffac8?, 0xc00022c1e0?}, 0x6?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3340 +0x8e
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).serve(0xc0000de480, {0x7fff70, 0xc0000c2ed0})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2109 +0x665
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*Server).Serve in goroutine 1
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3493 +0x485
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 191 [IO wait]:
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.runtime_pollWait(0x7fc97605a600, 0x72)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/netpoll.go:351 +0x85
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).wait(0xc00021a200?, 0xc000346561?, 0x0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).waitRead(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*FD).Read(0xc00021a200, {0xc000346561, 0x1, 0x1})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_unix.go:165 +0x279
Jan 31 07:16:18 xxxxx icons[8862]: net.(*netFD).Read(0xc00021a200, {0xc000346561?, 0xa2fbc0?, 0xc000082f70?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/fd_posix.go:68 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: net.(*conn).Read(0xc00030e058, {0xc000346561?, 0x0?, 0x0?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/net.go:196 +0x45
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*connReader).backgroundRead(0xc000346540)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:702 +0x33
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*connReader).startBackgroundRead in goroutine 657
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:698 +0xb6
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 612 [IO wait]:
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.runtime_pollWait(0x7fc976019c00, 0x72)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/netpoll.go:351 +0x85
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).wait(0xc00021a280?, 0xc0003465e1?, 0x0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).waitRead(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*FD).Read(0xc00021a280, {0xc0003465e1, 0x1, 0x1})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_unix.go:165 +0x279
Jan 31 07:16:18 xxxxx icons[8862]: net.(*netFD).Read(0xc00021a280, {0xc0003465e1?, 0xa2fbc0?, 0xc000082770?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/fd_posix.go:68 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: net.(*conn).Read(0xc00030e060, {0xc0003465e1?, 0x0?, 0x0?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/net.go:196 +0x45
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*connReader).backgroundRead(0xc0003465c0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:702 +0x33
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*connReader).startBackgroundRead in goroutine 658
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:698 +0xb6
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 659 [sync.Mutex.Lock]:
Jan 31 07:16:18 xxxxx icons[8862]: internal/sync.runtime_SemacquireMutex(0x41b691?, 0x98?, 0xd?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/sema.go:95 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: internal/sync.(*Mutex).lockSlow(0xc0000c2ba8)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/sync/mutex.go:149 +0x15d
Jan 31 07:16:18 xxxxx icons[8862]: internal/sync.(*Mutex).Lock(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/sync/mutex.go:70
Jan 31 07:16:18 xxxxx icons[8862]: sync.(*Mutex).Lock(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/sync/mutex.go:46
Jan 31 07:16:18 xxxxx icons[8862]: sync.(*RWMutex).Lock(0xa611a0?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/sync/rwmutex.go:150 +0x32
Jan 31 07:16:18 xxxxx icons[8862]: main.(*Cache).Set(0xc0000c2ba0, {0xc0002564b0, 0x10}, {0xc000912000, 0x4996})
Jan 31 07:16:18 xxxxx icons[8862]:         /app/main.go:65 +0x65
Jan 31 07:16:18 xxxxx icons[8862]: main.handleIcon({0x7ffac8, 0xc00044c1e0}, 0xc00064a640)
Jan 31 07:16:18 xxxxx icons[8862]:         /app/main.go:328 +0xf59
Jan 31 07:16:18 xxxxx icons[8862]: net/http.HandlerFunc.ServeHTTP(0xc0000e8000?, {0x7ffac8?, 0xc00044c1e0?}, 0x63e136?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2322 +0x29
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*ServeMux).ServeHTTP(0x473019?, {0x7ffac8, 0xc00044c1e0}, 0xc00064a640)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2861 +0x1c7
Jan 31 07:16:18 xxxxx icons[8862]: net/http.serverHandler.ServeHTTP({0xc000346640?}, {0x7ffac8?, 0xc00044c1e0?}, 0x6?)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3340 +0x8e
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).serve(0xc0000de5a0, {0x7fff70, 0xc0000c2ed0})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2109 +0x665
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*Server).Serve in goroutine 1
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3493 +0x485
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 660 [IO wait]:
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.runtime_pollWait(0x7fc976019e00, 0x72)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/netpoll.go:351 +0x85
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).wait(0xc00021a380?, 0xc00014e000?, 0x0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).waitRead(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*FD).Read(0xc00021a380, {0xc00014e000, 0x1000, 0x1000})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_unix.go:165 +0x279
Jan 31 07:16:18 xxxxx icons[8862]: net.(*netFD).Read(0xc00021a380, {0xc00014e000?, 0x7fc975cf3301?, 0xc0005086d0?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/fd_posix.go:68 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: net.(*conn).Read(0xc00030e070, {0xc00014e000?, 0xc0000c2ba8?, 0xc0002e0000?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/net.go:196 +0x45
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*connReader).Read(0xc0003466c0, {0xc00014e000, 0x1000, 0x1000})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:812 +0x154
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).fill(0xc0007160c0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:113 +0x103
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).ReadSlice(0xc0007160c0, 0xa)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:380 +0x29
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).ReadLine(0xc0007160c0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:409 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: net/textproto.(*Reader).readLineSlice(0xc0003bc240, 0xffffffffffffffff)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/textproto/reader.go:64 +0xa5
Jan 31 07:16:18 xxxxx icons[8862]: net/textproto.(*Reader).ReadLine(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/textproto/reader.go:44
Jan 31 07:16:18 xxxxx icons[8862]: net/http.readRequest(0xc0007160c0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/request.go:1087 +0xa5
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).readRequest(0xc0000de630, {0x7fffa8, 0xc000546460})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:1041 +0x227
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).serve(0xc0000de630, {0x7fff70, 0xc0000c2ed0})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2031 +0x3d8
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*Server).Serve in goroutine 1
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3493 +0x485
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 661 [IO wait]:
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.runtime_pollWait(0x7fc97605aa00, 0x72)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/netpoll.go:351 +0x85
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).wait(0xc00021a400?, 0xc0001da000?, 0x0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).waitRead(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*FD).Read(0xc00021a400, {0xc0001da000, 0x1000, 0x1000})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_unix.go:165 +0x279
Jan 31 07:16:18 xxxxx icons[8862]: net.(*netFD).Read(0xc00021a400, {0xc0001da000?, 0x7fc976011501?, 0xc0005096d0?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/fd_posix.go:68 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: net.(*conn).Read(0xc00030e078, {0xc0001da000?, 0xc0000c2ba8?, 0xc0004f2000?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/net.go:196 +0x45
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*connReader).Read(0xc000346740, {0xc0001da000, 0x1000, 0x1000})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:812 +0x154
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).fill(0xc000100720)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:113 +0x103
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).ReadSlice(0xc000100720, 0xa)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:380 +0x29
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).ReadLine(0xc000100720)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:409 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: net/textproto.(*Reader).readLineSlice(0xc0000c3560, 0xffffffffffffffff)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/textproto/reader.go:64 +0xa5
Jan 31 07:16:18 xxxxx icons[8862]: net/textproto.(*Reader).ReadLine(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/textproto/reader.go:44
Jan 31 07:16:18 xxxxx icons[8862]: net/http.readRequest(0xc000100720)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/request.go:1087 +0xa5
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).readRequest(0xc0000de6c0, {0x7fffa8, 0xc0005464b0})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:1041 +0x227
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).serve(0xc0000de6c0, {0x7fff70, 0xc0000c2ed0})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2031 +0x3d8
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*Server).Serve in goroutine 1
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3493 +0x485
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 662 [IO wait]:
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.runtime_pollWait(0x7fc976019a00, 0x72)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/netpoll.go:351 +0x85
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).wait(0xc00021a480?, 0xc000154000?, 0x0)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*pollDesc).waitRead(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
Jan 31 07:16:18 xxxxx icons[8862]: internal/poll.(*FD).Read(0xc00021a480, {0xc000154000, 0x1000, 0x1000})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/internal/poll/fd_unix.go:165 +0x279
Jan 31 07:16:18 xxxxx icons[8862]: net.(*netFD).Read(0xc00021a480, {0xc000154000?, 0x1?, 0xc0005986d0?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/fd_posix.go:68 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: net.(*conn).Read(0xc00030e080, {0xc000154000?, 0x0?, 0x0?})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/net.go:196 +0x45
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*connReader).Read(0xc0003467c0, {0xc000154000, 0x1000, 0x1000})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:812 +0x154
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).fill(0xc000716120)
Jan 31 07:16:18 xxxxx conmon[8862]: conmon 110b6d6dd39d27fc8c63 <nwarn>: Failed to open cgroups file: /sys/fs/cgroup/memory.events
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:113 +0x103
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).ReadSlice(0xc000716120, 0xa)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:380 +0x29
Jan 31 07:16:18 xxxxx icons[8862]: bufio.(*Reader).ReadLine(0xc000716120)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/bufio/bufio.go:409 +0x25
Jan 31 07:16:18 xxxxx icons[8862]: net/textproto.(*Reader).readLineSlice(0xc0000c3620, 0xffffffffffffffff)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/textproto/reader.go:64 +0xa5
Jan 31 07:16:18 xxxxx icons[8862]: net/textproto.(*Reader).ReadLine(...)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/textproto/reader.go:44
Jan 31 07:16:18 xxxxx icons[8862]: net/http.readRequest(0xc000716120)
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/request.go:1087 +0xa5
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).readRequest(0xc0000de750, {0x7fffa8, 0xc000546500})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:1041 +0x227
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*conn).serve(0xc0000de750, {0x7fff70, 0xc0000c2ed0})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:2031 +0x3d8
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*Server).Serve in goroutine 1
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3493 +0x485
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 663 [runnable]:
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*connReader).startBackgroundRead.gowrap2()
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:698
Jan 31 07:16:18 xxxxx icons[8862]: runtime.goexit({})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/asm_amd64.s:1693 +0x1
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*connReader).startBackgroundRead in goroutine 659
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:698 +0xb6
Jan 31 07:16:18 xxxxx icons[8862]:
Jan 31 07:16:18 xxxxx icons[8862]: goroutine 673 [runnable]:
Jan 31 07:16:18 xxxxx icons[8862]: net/http.(*Server).Serve.gowrap3()
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3493
Jan 31 07:16:18 xxxxx icons[8862]: runtime.goexit({})
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/runtime/asm_amd64.s:1693 +0x1
Jan 31 07:16:18 xxxxx icons[8862]: created by net/http.(*Server).Serve in goroutine 1
Jan 31 07:16:18 xxxxx icons[8862]:         /usr/local/go/src/net/http/server.go:3493 +0x485

[shollyethan]

Hi, thanks for the details. This seems to be caused by delete(c.items, key) below in main.go, which was an attempt to keep the cache up-to-date but creates mapping issues when something is deleted while multiple routines are concurrently reading from it:

if time.Since(item.Timestamp) > c.ttl {                                                                                                              
  delete(c.items, key)                                                                                                                               
  return "", false                                                                                                                                   
}

There are other mechanisms in place to the cache at bay, so I've removed it and have pushed a temp image to the registry with the fix if you wouldn't mind testing it before I push it out as a formal release:

ghcr.io/selfhst/icons:read-lock-test

[sreedevk]

Hi @shollyethan
I appreciate the quick response and fix. I've deployed the ghcr.io/selfhst/icons:read-lock-test image. I'll let it run for a few hours and report back.

[sreedevk]

@shollyethan I've run it for over an hour now and I tried sending a couple of hundred requests to the server in short bursts several times. I can confirm that I no longer see the concurrent map writes error in ghcr.io/selfhst/icons:read-lock-test. I also checked the docker stats for the container after the run, they all look normal as well.

ID            NAME        CPU %       MEM USAGE / LIMIT  MEM %       NET IO             BLOCK IO      PIDS        CPU TIME    AVG CPU %
0a2894be6e1e  icons       0.00%       10.17MB / 98.81GB  0.01%       6.552MB / 123.3MB  2.777MB / 0B  25          1.170403s   0.02%

Thank you again for looking into the issue and resolving it. I'll switch over to ghcr.io/selfhst/icons:latest once the fix has been made available there.