Temporary error when realizing layers during indexing leaves IndexReport in an unrecoverable error state

[frostmar]

When indexing a digest, an error during "realizing layers" (ie. fetching and unpacking the tar-gz layer data) will cause IndexReport to be persisted with an error state into Postgres. That state is then persistent - claircore won't reattempt indexing even if another index request is made.

It's possible to cludge in a workaround that inspects the index response or fetches an IndexReport after indexing, then deletes the report for that digest before retrying ... but this is clumsy, couples to the internals of claircore, and has less information about the error to work with.

---

Example logs

From memory I think in this case the IndexReport ended up with state IndexError.

message:index request start

message:locking attempt

message:locking OK

message:starting scan

message:manifest to be scanned

message:layers fetch start

message:fetching layers

message:layer fetch start

... other layer fetch starts snipped ...

message:layers fetch failure
  error:fetcher: encountered errors: error realizing layer sha256:9dc6ea54f2e2297d66090e7d724af8a0648caef7d7fc589a3da7433470bd9100: unexpected status code: 503 Service Unavailable (body starts: "upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused")

message:layers fetch done
  state:FetchLayers

message:error during scan
  error:failed to fetch layers: fetcher: encountered errors: error realizing layer sha256:9dc6ea54f2e2297d66090e7d724af8a0648caef7d7fc589a3da7433470bd9100: unexpected status code: 503 Service Unavailable (body starts: "upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused")

message:index request done

message:http error response
  code:500
  error:failed to start scan: failed to fetch layers: fetch

message:handled HTTP request

[frostmar]

This is one specific example where errors that can be temporary/intermittent, and would likely be resolved with a retry, are persisted in a way that can't easily be recovered from.

A similar type of issue around recovering from intermittent errors is:

• Timeout while rescanning a previously scanned manifest leaves IndexReport empty forever #1124

---

It feels like indexing would benefit from a more in-depth redesign around failure states. Maybe have two classes:

1. "permanent errors"
   Anything that's impossible to change on a retry of indexing eg. files found in unrecognized formats
2. "potential recoverable errors"
   Every other error, to cover things like out of disk temp space / any http 5xx error etc