feat: add checkpointing into the backfill so we can resume from the l…

[knrc]

…ast completed, contiguous index

Closes #2279

Summary

This pull request adds support for checkpointing the progress of the backfill process, allowing the process to resume from the last known, contiguous index. This addresses the issue of the backfill reprocessing information from the beginning each time it runs.

Release Note

Added support for checkpointing progress during the backfill operation, rerunning the backfill will resume from the last known, contiguous index instead of starting at the beginning.

Documentation

[codecov]

Codecov Report

❌ Patch coverage is 0% with 158 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.03%. Comparing base (488eb97) to head (63b3c04).
⚠️ Report is 650 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/backfill-index/main.go	0.00%	158 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2762       +/-   ##
===========================================
- Coverage   66.46%   26.03%   -40.43%     
===========================================
  Files          92      191       +99     
  Lines        9258    20236    +10978     
===========================================
- Hits         6153     5269      -884     
- Misses       2359    14138    +11779     
- Partials      746      829       +83     

Flag	Coverage Δ
e2etests	49.49% <ø> (+1.93%)	⬆️
unittests	16.63% <0.00%> (-31.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Hayden-IO]

@knrc Could we simply write a log index periodically to disk to simplify this? Backfilling is idempotent, so we don't need to be overly precise with the latest index.

[knrc]

@knrc Could we simply write a log index periodically to disk to simplify this? Backfilling is idempotent, so we don't need to be overly precise with the latest index.

Yes, that's possible but I felt it would be better to keep the checkpoint within the same storage so they are managed together. The code is already taking advantage of the idempotency when tracking/updating the checkpoint, and does expect to replay on out of sequence updates or errors.

[Hayden-IO]

Given the amount of code needed for this, I'd strongly prefer a much simpler solution, especially given the shift to Rekor v2.