Support custom actions on service exit events

[jy5275]

In SONiC on Ubuntu, the service exit behavior is stored in an external configuration database.
When a service exits, supervisord captures that status change event and triggers a python script, which then reads the database to decide what to do next, such as restart, ignore, shutdown or do other custom logic.

Currently, Pebble only supports a fixed set of post-exit behaviors via on-success or on-failure options:

• restart
• ignore
• shutdown
• failure-shutdown

We would like Pebble to support custom logic execution upon service exit events — similar to Supervisord’s event system (Events — Supervisor documentation ).

Pebble’s existing Notices feature provides a somewhat similar mechanism, but it focuses on engine-level changes, not service-level events.

We want a more dynamic and flexible event mechansim for service status changes behavior.

[benhoyt]

Note for Charm Tech: this is somewhat related to #104, where we wanted better support for introspecting service failures. One way to do that would be to add a Change/Task for service "restarting" or "backoff" and then you could watch change-update notices to detect this. Alternatively we could add a new, more specific notice type like service-exited.

@jy5275 If we do decide to do this "Pebble notices", you can listen for new notices via the API. However, that would require a little background service that's watching for notices (say using the Python Pebble API in ops.pebble) and then runs the custom logic you want to when it gets updates. Would this work for you?

[benhoyt]

This is now on our 26.04 roadmap. We still need to write a spec (and discuss with Gustavo) as there are a couple of different approaches.

[jy5275]

Note for Charm Tech: this is somewhat related to #104, where we wanted better support for introspecting service failures. One way to do that would be to add a Change/Task for service "restarting" or "backoff" and then you could watch change-update notices to detect this. Alternatively we could add a new, more specific notice type like service-exited.

@jy5275 If we do decide to do this "Pebble notices", you can listen for new notices via the API. However, that would require a little background service that's watching for notices (say using the Python Pebble API in ops.pebble) and then runs the custom logic you want to when it gets updates. Would this work for you?

Hi Ben, that works for us. Will that "background service" managed inside by pebble or outside pebble? I think the former approach could make it more convenient to manage all processes in a rock, but may cause "endless" notices by the custom listener itself if it's buggy.