Add STH caching with configurable timeouts and intelligent retry

[bobcallaway]

This PR adds an optional cached Signed Tree Head (STH) path to the Trillian client to reduce repeated root-fetch overhead on hot paths while preserving correctness guarantees.

When --trillian_log_server.cache_sth=True, the client maintains a background root updater, stores the latest verified root for fast concurrent reads, and wakes waiting callers when the tree advances. The non-cached behavior remains available as the default path.

What changed

• Added a cache-enabled Trillian client implementation with background root updates.
• Added a simple stateless client implementation and interface-based selection.
• Wired server/config plumbing for the cache_sth feature flag.
• Updated manager lifecycle/close behavior for safe client reuse and shutdown.
• Added focused tests for updater retries, manager caching/close semantics, and simple-client behavior.
• Added load-test scaffolding for evaluating throughput/latency impact.

Why

The previous flow repeatedly fetched/validated roots during high concurrency, creating avoidable latency and contention. Caching the latest verified root allows callers to read quickly while still reacting to tree advancement.

Impact

• Lower per-request overhead on read/write paths under load.
• Better scaling behavior when many callers contend on the same tree.
• Performance improvements are observable in local/load-test runs, but exact gains depend on deployment topology and workload mix.

Safety / rollout

• Feature is gated behind trillian_log_server.cache_sth (experimental) command line argument.
• Existing behavior is preserved when the flag is disabled.
• Includes test coverage for concurrency and transient update failures.

[codecov]

Codecov Report

❌ Patch coverage is 65.77540% with 192 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.93%. Comparing base (488eb97) to head (353516a).
⚠️ Report is 637 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/trillianclient/trillian_client_simple.go	60.75%	72 Missing and 21 partials ⚠️
pkg/trillianclient/trillian_client.go	67.50%	76 Missing and 15 partials ⚠️
pkg/trillianclient/manager.go	85.18%	2 Missing and 2 partials ⚠️
pkg/api/api.go	77.77%	1 Missing and 1 partial ⚠️
pkg/trillianclient/client_interface.go	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2583       +/-   ##
===========================================
- Coverage   66.46%   26.93%   -39.53%     
===========================================
  Files          92      193      +101     
  Lines        9258    20583    +11325     
===========================================
- Hits         6153     5544      -609     
- Misses       2359    14179    +11820     
- Partials      746      860      +114     

Flag	Coverage Δ
e2etests	47.16% <32.62%> (-0.40%)	⬇️
unittests	18.01% <44.56%> (-29.68%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.