True zero-downtime deployment by request draining

[wowu]

Currently there's no way of telling Traefik that old container is going to be stopped, so it might route requests to a container that is shutting down. I'm creating this issue to track the progress of figuring out what is the best way of implementing this.

The problem was mentioned in this StackOverflow question: https://stackoverflow.com/questions/75918681/how-to-avoid-downtime-when-using-docker-rollout-with-traefik

Current idea

The easiest way seems to fail healthchecks before the container is going to be stopped, so Traefik is not routing new requests to the unhealthy container(s). This can be achieved by adding ! test -f /drain to container healthcheck, that is "fail if there exists a file named drain in /", and docker-rollout can create this file before stopping the old container.

I'm not sure if this behavior should be hardcoded in the tool, as there might be better ways of implementing request draining for proxies other than Traefik / nginx. Implementing hook support would allow docker-rollout users to implement true zero downtime deployment in two steps:

1. Add && ! test -f /drain to current container healtcheck in compose file
2. Add a hook like --before-stop "docker exec $1 touch /drain && sleep 10" to create the file manually

[rogerdz]

https://doc.traefik.io/traefik/middlewares/http/retry/
maybe this can help ?

[immortaly007]

I like the current idea, but wouldn't the "sleep" timout need to be at least the interval of the health check times the "retry" amount for the health check? Otherwise the container might not enter the "unhealthy" state before the shutdown sequence starts.

Maybe another option (specifically for traefik): in the --before-stop hook, add the label traefik.enable=false. (note that I have not tested how quickly traefik would pick up this change).

[wowu]

Thats a useful insight, thanks! We should describe the requirements for sleep in docs if we go for the hooks solution, or wait a required time in docker-rollout itself.

Docker container labels are immutable (at least for now: moby/moby#21721), so we cannot use them to deregister the container from Traefik.

[wowu]

I created a sample implementation in #36, it would be amazing if someone could test it with their setup to confirm it works correctly 😄

[ku9nov]

Hello @wowu
First of all, I want to express my gratitude for this project.

As I see it, this issue hasn't been progressing on your end. A few days ago, I decided to explore this topic to implement it for my small projects that don’t use Kubernetes or other similar tools since they are relatively small.

Upon discovering your project, I realized it was exactly what I needed. However, we all know about the problem with draining connections in proxy servers.

After some experimentation, I came up with my own solution to this issue GitHub Repository. While it is inspired by your project, it has significant differences, so I didn’t fork it but instead implemented everything separately for now. My main priority is easy integration into existing projects, and for these reasons (which are part of my plans):

1. Proxy containers will be included by default for this plugin.
2. For Traefik, a dynamic configuration will be used, which will be automatically generated by the plugin based on container labels.
3. A solution for nginx-proxy is also planned.
4. In addition to rolling deployment, I also plan to implement settings for blue-green and canary deployments.

Other ideas as well.

My goal is to make everything fully automated and work out-of-the-box, so I hope you understand why it currently looks like this.

The project is already deployable and can be easily tested with a single command (achieving zero-time deployment):
GitHub Repository

Please note that this has not yet been merged into the main branch, so you need to use this specific branch.

These are my needs, and the changes are quite substantial, so they might not align with your vision. However, I’m writing this because I respect other people's work and the open-source community. If you're interested in reworking the project to utilize proxy servers in an automated way and implementing this approach, I can convert it into a fork, and we can collaborate on it.

[wowu]

Finally deployed in https://github.com/wowu/docker-rollout/releases/tag/v0.12. Also configurable via a docker-rollout.pre-stop-hook container label.