Support for other signals beside SIGTERM

[oktalz]

When service is planned to be stopped, pebble sends a SIGTERM signal,

this is visible on two places (if I'm not mistaken)

pebble/internals/overlord/servstate/handlers.go

Line 731 in d2ca348

err := syscall.Kill(-s.cmd.Process.Pid, syscall.SIGTERM)

pebble/internals/overlord/servstate/handlers.go

Line 862 in d2ca348

err := syscall.Kill(-s.cmd.Process.Pid, syscall.SIGTERM)

this is perfectly acceptable, but, would it be possible to customize this ?

for example setup could look something like this:

services:

    my-app:
        override: replace
        command: /path/my-command.sh
        startup: enabled
        on-success: ignore
        on-failure: restart
        kill-delay: 30s
        stop-signal: SIGUSR1

if stop-signal is defined, that signal is used for stop, otherwise, SIGTERM is used

to explain, while SIGTERM works and it closes application, some applications supports graceful shutdown through other signals like SIGUSR1.
For example HAProxy is one of them. if SIGTERM is issued, application will exit, but it will not gracefully wait until all connection are done and that is not ideal.

would something like this be acceptable to implement as an additional feature ?

[benhoyt]

Thanks for the suggestion. We have a pretty full roadmap, but we'll consider this. For (our) reference, that HAProxy feature is documented here.

Out of interest, what are you using Pebble for? Can you describe your use case in a bit more detail?

[oktalz]

Out of interest, what are you using Pebble for? Can you describe your use case in a bit more detail?

Sure, but I can give higher amount of info next week when we push project to public :).

In short, we have several docker containers that are using some kind of init system because we need to run two processes in single container. We can discuss if that is a good approach or not, but that is another topic.
Currently when needed, we used s6 overlay for this purpose, however recently that is becoming more and more of an issue. Reason why this is concern to us is that we have containers that are rootless and with s6 this does not work ideally in k8s environment (you can find enough data about it if you search for S6_YES_I_WANT_A_WORLD_WRITABLE_RUN_BECAUSE_KUBERNETES), but, as you can see from ENV flag, this is something that maintainers of s6 do not really prefer. While it is manageable, s6 dumps a lot of messages on start that seems like an errors.

For a longer time now (from 2023) we offer for one of our projects (Kubernetes Ingress Controller), a docker image with pebble, but as you will see, we were not really pushing it and it was more as an experiment to see how it behaves and to gather some data around it. This might get an upgrade soon too.

However, with our new open source project, HAProxy Unified Gateway - HUG (there is no link since release will happen next week, I'll provide one here as that happens) we opted to use pebble by default. This is ideal for us, since first release will be a beta so if something goes wrong, we can fallback to different solutions.

During testing, basically we figured that pebble is missing a feature to cover HAProxy graceful shutdown, and hence, this feat request (I also added PR since I wanted to see how doable is that).

while container is shutting down, it is really not ideal that HAProxy just kills open connections, we need to send SIGUSR1 to trigger graceful stop.

one might think to use k8s preStop hook (one approach)

          lifecycle:
            preStop:
              exec:
                # Send SIGUSR1 directly to the haproxy process
                command: ["/bin/sh", "-c", "kill -SIGUSR1 $(cat /var/run/haproxy.pid);"]

but that is not really a solution, in that case we would need to wait additionally to this to stop and then proceed. So, it is doable to write something like that, but it is not elegant.

I hope this explains a bit why I need that or similar feature.
I'm also willing to contribute (PR :) ), but I'm absolutely fine if you decide to implement this or something similar in your own way,
but I would really like to see that feature in pebble if possible.

[benhoyt]

Thanks for the additional context. We'll consider this feature in the next few weeks and update this issue.

[dimaqq]

Hi @oktalz, is this in the context of Juju, or rocks being run directly on Docker or other similar system?

[oktalz]

@dimaqq I run it in k8s in docker, though I do have same issue also on barebone machine (but that is more of a hobby project)

[tonyandrewmeyer]

some applications supports graceful shutdown through other signals like SIGUSR1. For example HAProxy is one of them.

Are you aware of any other examples? In my experience, SIGUSR1 is commonly used for logging adjustments, config reloading, dumping state, and that sort of thing. OpenVPN will restart on SIGUSR1 if needed, I believe, but that's the closest example I can think of outside of HAProxy.

Even the HAProxy docs imply that SIGTERM is the right one for "stop", rather than "restart":

The hard stop method is used for the "stop" or "restart" actions of the service management script. The graceful stop is used for the "reload" action which tries to seamlessly reload a new configuration in a new process.

By the way, if you want to send a signal, you can do that through Pebble rather than using kill, using pebble signal. If I understand correctly, HAProxy will exit after all the connections are done, so couldn't you just pebble signal rather than pebble stop?

[oktalz]

Are you aware of any other examples? In my experience, SIGUSR1 is commonly used for logging adjustments, config reloading, dumping state, and that sort of thing

yes, nowadays, but that was not the case always. and yes SIGUSR is basically user signal for whatever.
some more examples (more known applications):

• Consul uses SIGINT to do a graceful termination, SIGTERM just kill the agent
• Nginx uses SIGQUIT for same reason (SIGTERM also is not desired to do if you want graceful shutown)
• Redis does use SIGTERM but also has SIGINT (for same purpose but no save on exit - that might be desired in some cases)
• many old java aplications use SIGINT, since SIGTERM was not a standard for that purpose

I would say this is especially important in docker environments.

Even the HAProxy docs imply that SIGTERM is the right one for "stop", rather than "restart":

HAProxy has many ways to reload and curently the best one is by using socket, not signals (then you get feedback), but this is irrelevant here. its called hard stop for a reason, because its not graceful.

If you have traffic that have long running connections (web downloads, web sockets etc..) its not ideal to immediately terminate.
with SIGUSR1 HAPRoxy stops serving new connections, but allows open ones to end.
this is ideal in k8s environment where POD is marked for deletion, we stop new traffic (healthchecks etc, but still allow for some time to handle existing connections)

HAProxy will exit after all the connections are done, so couldn't you just pebble signal rather than pebble stop?

I'm not triggering anything, k8s is, when k8s wants to shutdown POD. it is fully automatic.
problem is even if i use something else to send the correct signal, pebble will send SIGTERM immediately.

[oktalz]

just for sake of having extra information, this is the project where I need that feature:
github.com/haproxytech/haproxy-unified-gateway and also config, so it is a pretty simple setup (its not complete, but it can be)

for beta that is currently active we can survive without signal override.

[benhoyt]

Thanks for the additional info. Just to set expectations: we've discussed this as a team, and it's likely we won't do this. I'll keep it open so we can consider it in more depth later, however.

However, I notice from your config that you're already using wrapper scripts to run the HAProxy binary. So the simplest thing would be to have the run-haproxy script catch SIGTERM and send SIGUSR1 to HAProxy instead. Would that work?

[oktalz]

thx for the info. I will monitor it.

However, I notice from your config that you're already using wrapper scripts to run the HAProxy binary. So the simplest thing would be to have the run-haproxy script catch SIGTERM and send SIGUSR1 to HAProxy instead. Would that work?

simply put, no.
wrapper script is something that is there to limit memory usage, even for that is seems like we need service that could do that. bringing more just means that we over-complicate design.
We can do many things in many ways, but we kind of opted to have a dedicated init system in container. any kind of bash scripting is not really preferred (even the one we have now)

similar functionalities do exist in systemD (that is not really a good candidate for docker containers) and S6 (that kind of works), so my hope was that this could seem like a good idea here too. Also, Pebble as a service manager is a good candidate for that purpose in containers.