Unnecessary word of caution: Explicitly only for the test cases involving cryptographic code paths.
An example of this might include an end-to-end "integration" test with a generated QR image as source material. These test images can be pre-generated—and are unlikely to change—with only the public key shipped with the codebase for validation, or a tool such as git-crypt can be used to protect the private key in use. I've used this with good effect in the past, and it integrates using GPG, thus works well if code signing will be utilized anyway and can support multi-person teams.