transient failure building due to unresolved derivations when building ca-derivations with an s3 store

[benaryorg]

Describe the bug

hydra-queue-runner: performing step ‘/nix/store/6n8l16fjlxflyl1q8da7hs3z7nfyji9b-xtrans-1.5.0.drv’ 1 times on ‘nix-ssh@[builder]’ (needed by build 11274 and 1 others)
hydra-queue-runner: possibly transient failure building ‘/nix/store/6n8l16fjlxflyl1q8da7hs3z7nfyji9b-xtrans-1.5.0.drv’ on ‘nix-ssh@[builder]’: error: the derivation '/nix/store/6n8l16fjlxflyl1q8da7hs3z7nfyji9b-xtrans-1.5.0.drv' can’t be resolved. It’s probably missing some outputs

Hydra itself is configured using a store_uri pointing at an S3 bucket (self-hosted, Ceph radosgw) which is publicly readable.
The same bucket (and associated public keys for the signatures) is added to the substituters on the builder.
Nix on the hydra node has builders-use-substitutes = true and hydra itself has useSubstitutes = true.
The drv which cannot be found on the builder can be pulled from the binary cache using nix copy --from just fine, although I cannot confirm whether the drv is present on the bucket at the time of the error.
The drv is also present in the /nix/store of the hydra node.
This only happens in builds using ca-derivations and was already mentioned in a related PR, however the PR does not contain any related changes (at this time) as this was tested against commits present in master.

It looks like hydra is neither copying the drv over manually, nor is the builder trying to fetch it.

To Reproduce
Steps to reproduce the behavior:

1. run hydra with store_uri of an S3 bucket
2. configure only remote builders for hydra
3. build a ca-derivations build
4. ???
5. these errors show up in the logs and the (not quite to CA adjusted) UI shows errors and a lot of blanks

Expected behavior
The build succeeds.

Hydra Server:

Please fill out this data as well as you can, but don't worry if you can't -- just do your best.

• OS and version: NixOS 24.05

hydra version	hydra.package.override { nix = ...; }	hydra nix.package	buildhost nix.package
879ceb5	Nix 2.21	Lix 2.90.0	Lix 2.90.0
879ceb5	Nix 2.21	Nix 2.21	Lix 2.90.0
879ceb5	Nix 2.21	Nix 2.21	Nix 2.21
d798622	Nix 2.22	Lix 2.90.0	Lix 2.90.0

Additional context
All other builds seem to run normally. It may be the low volume of other builds, but it genuinely looks like only the ca-derivations builds have this issue.

[gador]

Same for me. Also tested with the same PR and trying to build a CA-enabled configuration. Other CA derivation (packages) build fine. The missing drv is present on the hydra node and on the builder