Configure units to automatically restart under certain failure conditions

[wallrj]

We'd like to be able to configure units to automatically restart under
certain failure conditions.

• http://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=

Eg

• If a user deliberately stops the docker container associated with a
  gear unit.
• Or if the docker container image takes so long to download that
  the systemd unit times out and enters the failed state. (although that
  may be an issue that we should address by pre-caching the docker image
  before calling gear install --start

That last problem may also be addressed by the docker readiness
notification idea in this PR
#175

The idea of recommending the systemd restart feature by default is
described here:

• https://fedorahosted.org/fpc/ticket/191

See:

• Original mailing list discussion at http://lists.openshift.redhat.com/openshift-archives/users/2014-July/msg00026.html
• An example of how this affects us in flocker: "Failed" gear units (applications) are not restarted  ClusterHQ/flocker#326 (comment)

[smarterclayton]

Is there a max restart limit that would make sense for all containers (ie something that would be a good default)?

[wallrj]

Sorry for delayed reply.

I can't see a "max restart limit" option. There's http://www.freedesktop.org/software/systemd/man/systemd.service.html#StartLimitInterval=

Supervisord sets a default maximum restart limit of 3 http://supervisord.org/configuration.html#program-x-section-settings

Twisted Procmon doesn't implement a max restart limit; it just allows you to limit the rate with a backoff algorithm. https://twistedmatrix.com/documents/current/api/twisted.runner.procmon.ProcessMonitor.html

Also worth noting that Docker (1.2.0) now has a "restart policy" feature which continues to attempt restarts indefinitely....but with a backoff:

• Proposal: Auto restart containers moby/moby#7226
• Auto restart containers based on restart policy moby/moby#7414