Watcher crash loop generates bursts of ethereum calls

[unnawut]

Overview

When a watcher is misconfigured or a bug occurs, the watcher could enter a crash loop and restarts all the processes within. This in turn causes a burst of Ethereum calls to start all the processes. And because the crash is not self-recoverable, the burst could happen every few seconds (5 seconds?). This impacts integrators who relies on Ethereum node as a service e.g. Infura, and causes them to exhaust the quota rapidly.

Actual behaviour

When the watcher is misconfigured or a bug occurs, it brings the watcher into an irrecoverable crash loop and produces a constant burst of Ethereum calls every few seconds.

Expected behaviour

When the watcher is misconfigured or a bug occurs, the watcher should have an incremental backoff period for retries.

[InoMurko]

This is quite a concerning issues. Did this ever occur? Is there a way to replicate this? Have you seen it in the wild and can provide logs?