Example for Caddy

[FluffyDiscord]

Hi, would it possible to add example for Caddy? Traefik is a bit configuration heavy and Nginx-proxy isn't secure because it's mounting docker socket

[mbrodala]

At least for the latter (and Traefik) you could use a docker socket proxy.

[jobbanens]

I use this for my Caddyfile:

:80 {
    # Route to first healthy backend instance found
    reverse_proxy backend:3000 {
        lb_policy first
    }

    # No instances available? Respond with an error
    handle_errors {
        respond "Service Unavailable" 503
    }
} 

And this for my compose.yaml:

services:
  backend:
    build: .
    restart: unless-stopped
    healthcheck:
      test: ['CMD-SHELL', 'test ! -f /tmp/drain && curl -f http://localhost:3000/health']
      interval: 5s
      timeout: 5s
      retries: 3
      start_period: 60s

  backend-proxy:
    image: caddy:2.7-alpine
    restart: always
    ports:
      - '3000:80'
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro

[FluffyDiscord]

huh, is it really that simple?

[michie1]

@jobbanens Please note, from my understanding, if you specify lb_policy first it will pick the first healthy container, but if you use replicas the load will be unbalanced.

Another strategy is to use the random lb_policy, which is default, but configure a health check in Caddy. See https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#active-health-checks and https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#syntax.

edit: it's unclear for me how Caddy knows the list of containers related to a service and "pick" the first available one.

[jobbanens]

@michie1 Correct, didn't think about the case when you want to scale to multiple containers. Also, Caddy seems to automatically work without the health check specification, but it's probably good practice to add it anyway. Thanks for the tips!

[tribut]

As far as I understand it, the current implementation of request draining really only works with Traefik, because this is the only reverse proxy that honors the health status of docker containers.

Caddy has no way of knowing if a container is healthy or unhealthy to docker. Additionally, lb_policy only works when Caddy knows multiple backends, which is not the case with the above sample config. You can easily confirm this by enabling the debug global option.

What really happens is that Caddy hands the hostname of your service to net.Dial(). When scaling up a service, the hostname will resolve to multiple IPs. net.Dial() will then try each of the IPs consecutively, until a connection succeeds. That means that as soon as the old container stops responding to requests, Go will implicitly use the fallback (either immediately when the connection is rejected or after a timeout).

In order to explicitly control where Caddy sends traffic, you would need to use dynamic upstreams, like this:

reverse_proxy {
  dynamic a backend 3000 {
    refresh 10s # default is 1m
  }
}

This adds an upstream for each of the returned IPs, but still leaves the question of how to inform Caddy to avoid the old container. Active health checks are not supported for dynamic upstreams which means your backend would have to have some additional logic to either serve a special http response code or flat out reject new connections which Caddys passive health checks can use.

The real solution here would be for docker-compose to remove unhealthy containers from the DNS response with multiple replicas, but I neither know if this is possible nor can I find any indication that it is planned.