Add bounded timeouts to BlobStore shutdown join() calls

[joewiz]

Summary

BlobStoreImpl.normalClose() and abnormalPersistentWriterShutdown() called Thread.join() with no timeout on the PersistentWriter and BlobVacuum threads. If either thread failed to terminate promptly, the shutdown would hang indefinitely — causing CI test suite timeouts when the surefire fork couldn't exit.

This supersedes #6179, which incorrectly made BlobStore threads daemon threads. As @adamretter (author of BlobStore) correctly pointed out, these threads must remain non-daemon to ensure pending blob writes complete. The fix is bounded join() timeouts during shutdown — giving threads 30s to drain before proceeding.

What changed

BlobStoreImpl.java — Added 30-second bounded timeouts to all three Thread.join() calls:

• normalClose() — PersistentWriter: join(30_000), then interrupt() + join(5_000) if still alive
• normalClose() — BlobVacuum: join(30_000) with warning if still alive
• abnormalPersistentWriterShutdown() — BlobVacuum: join(30_000) with warning if still alive

Why this is safe

The existing shutdown mechanisms are correct and unchanged:

1. PersistentWriter receives a POISON_PILL via the persist queue → drains remaining items and exits
2. BlobVacuum receives interrupt() → wakes from PriorityBlockingQueue.take() and exits

In practice, both threads respond within milliseconds. The 30s timeout is a safety net — it only fires if something goes wrong, ensuring shutdown completes instead of hanging forever.

Test results

• 27/27 BlobStore-specific tests pass
• 3/3 full test suite trials: BUILD SUCCESS, clean exit (3:26–4:23)

Test plan

• BlobStoreImplTest — 27 tests, 0 failures
• Full mvn test -pl exist-core — 3 trials, all pass, no hangs
• CI passes on all platforms

🤖 Generated with Claude Code

[adamretter]

@joewiz Unfortunately, in practice this has the same corruption issue as your previous approach in #6179, it just goes abount achieving that corruption in a different manner.

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Closing this PR. Adam is right that the bounded timeout has the same fundamental risk as the daemon thread approach — if the timeout fires and closeBlobStore() proceeds while the writer is mid-write, data corruption is possible.

The CI hang symptom that motivated this fix is now addressed through other means:

• Fix BrokerPool shutdown hang causing CI timeouts #6167 (merged) and Fix ExecutorService thread leak in TransactionTestDSL #6175 (merged) fixed the dominant shutdown hangs — develop baseline is clean in our controlled experiment (5/5 trials, 0 hangs)
• [ci] Add surefire fork timeouts to prevent CI hangs #6186 adds surefire fork timeouts as a safety net, ensuring CI never burns 45+ minutes on a hung fork

The underlying BlobStore shutdown behavior (non-daemon threads that can prevent JVM exit if they hang) remains as-is. It's the correct design for data integrity — these threads must complete their work before the JVM exits.