flaky test: TestLibp2pACMEE2E timed out waiting for certificate

[lidel]

running GOTESTFLAGS=-shuffle=on go test -count=10 -failfast -race -v -p 1 -run TestLibp2pACMEE2E sometimes fails with: timed out waiting for certificate

usually its WithRegistrationDelay one, sometimes WithShortForgeAddrs

=== NAME  TestLibp2pACMEE2E/explicit_WithShortForgeAddrs(false)
    e2e_test.go:722: timed out waiting for certificate
POST /my-order/ -> calling handler()
POST /certZ/ -> calling handler()
GET /draft-ietf-acme-ari-03/renewalInfo/ -> calling handler()
=== NAME  TestLibp2pACMEE2E/WithRegistrationDelay()_produces_a_delay
    e2e_test.go:722: timed out waiting for certificate
--- FAIL: TestLibp2pACMEE2E (0.00s)
    --- PASS: TestLibp2pACMEE2E/explicit_WithShortForgeAddrs(true) (8.81s)
    --- PASS: TestLibp2pACMEE2E/default_opts (10.80s)
    --- PASS: TestLibp2pACMEE2E/expired_cert_gets_renewed_and_triggers_OnCertRenewed (21.89s)
    --- FAIL: TestLibp2pACMEE2E/explicit_WithShortForgeAddrs(false) (32.91s)
    --- FAIL: TestLibp2pACMEE2E/WithRegistrationDelay()_produces_a_delay (47.17s)

This may be related to panic:

2025/06/25 22:30:42 panic: certificate worker: runtime error: index out of range [0] with length 0
goroutine 5699 [running]:
github.com/caddyserver/certmagic.(*jobManager).worker.func1()
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/async.go:58 +0x6f
panic({0x2264a40?, 0xc0090c54d0?})
	/usr/lib/go/src/runtime/panic.go:792 +0x132
github.com/caddyserver/certmagic.(*ACMEIssuer).selectPreferredChain(0xc008f3c840, {0x0, 0x0, 0x0})
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:604 +0x52b
github.com/caddyserver/certmagic.(*ACMEIssuer).doIssue(0xc008f3c840, {0x2a6acf0, 0xc0075a96b0}, 0xc00cba3b88, 0x0)
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:523 +0x14c5
github.com/caddyserver/certmagic.(*ACMEIssuer).Issue(0xc008f3c840, {0x2a6acf0, 0xc0075a96b0}, 0xc00cba3b88)
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382 +0xed
github.com/caddyserver/certmagic.(*Config).obtainCert.func2({0x2a6acf0, 0xc0075a96b0})
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/config.go:626 +0x14b9
github.com/caddyserver/certmagic.doWithRetry({0x2a6ad28, 0xc00ac31e00}, 0xc008499100, 0xc006b2f8f0)
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/async.go:104 +0x2c3
github.com/caddyserver/certmagic.(*Config).obtainCert(0xc005dc8ee0, {0x2a6ad28, 0xc00ac31e00}, {0xc00b6d5860, 0x4e}, 0x0)
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/config.go:700 +0xa95
github.com/caddyserver/certmagic.(*Config).ObtainCertAsync(...)
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/config.go:505
github.com/caddyserver/certmagic.(*Config).manageOne.func1()
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/config.go:415 +0xb9
github.com/caddyserver/certmagic.(*jobManager).worker(0x37952a0)
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/async.go:73 +0x1b5
created by github.com/caddyserver/certmagic.(*jobManager).Submit in goroutine 4638
	/gopath/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/async.go:50 +0x426

Potential problem sources:

• all tests in TestLibp2pACMEE2E reuse global instance of p2p-forge backed by badger datastore initialized in TestMain, this creates surface for bugs
• certmagic tests use a mock ACME server called "pebble" (NOTE: different than "pebble" datastore we have in ipfs), could be a racy bug there