Building integration tests binaries as a derivations

[MiniaczQ]

Is your feature request related to a problem? Please describe.
I want to use crane for integration tests, but I can't run the integration tests in build phase. I want to export the binaries for later use in a more elaborate setup.

Describe the solution you'd like
Support exporting integration test binaries.
Here all test binaries are removed from the output.
I'd like a way to retain integration test binaries.
We can tell them apart through target.kind, unit tests being bin, integration tests being test.
I don't know how this would be configured

Additional context
I don't see an easy way of implementing this user-side.

#600 is loosely related.

[dpc]

FWIW. We run integration (/e2e?) tests with crane. You can trace it here: https://github.com/fedimint/fedimint/blob/d8afa208426a66b41db04d46b291c73833c4af3b/nix/flakebox.nix#L760

How it works, is basically building a whole workspace, and then using craneLib.mkCargoDerivation chainged via cargoArtifacts to run arbitrary commands (like cargo nextest, or our custom mega-meta-parallel-test-runner).

There are many reasons to do things exactly how we do it there. The core insight there is in the CI, you probably want to build everything all at once as a workspace, to save time and make it compress really well together (save space), and then you want to express your test script to work on the workspace (binaries inside .../target/<profile>) setup, so it is possible to run the same test script both normally in dev shell, and inside a custom Nix derivation with almost no difference between the two.

[MiniaczQ]

That's the thing, you seem to be running tests in the build phase:
https://github.com/fedimint/fedimint/blob/d8afa208426a66b41db04d46b291c73833c4af3b/nix/flakebox.nix#L748

I'm making a microservice and I need to setup surrounding infrastructure, ideally exact containers I use in production.
Best thing I could find was arion, which wraps docker compose pretty neatly, but it cannot run in a sandbox easily.

So I'd prefer to export the integration test binaries as derivation/-s, then create a test runner derivation which I can launch in a pipeline.

[dpc]

Oh I see.

I'm not sure how easy it will be to extract particular unit-test binares, but what we do is just build tests as normal binaries, which then can be built as a part of the workspace and extracted as normal, to run something outside of the sandbox. We like to build all/most e2e tests as subcommands of a single test-runner binary: https://github.com/fedimint/fedimint/blob/c44d554f6386d092a649c741aed1f83e9a82f60b/scripts/tests/latency-test.sh#L11 and call it. Though we do call it from a Nix sandbox currently, but it would work the same from outside of it.

Alternatively, you can build a workspace and extract it as a whole by either importing (somehow) and reussing:

crane/lib/setupHooks/inheritCargoArtifactsHook.sh

Line 1 in 2510f2c

inheritCargoArtifacts() {

, or re-implementing this functionality. Maybe there is an existing craneLib method that does it already. Or maybe there should be. I think at least once I needed it before and then I just implemented it from scratch.

There might be things here that crane could do to make things like this easier, but I'm just pointing out you should be able to unblock yourself with a bit of creativity and hacking.

[ipetkov]

Here all test binaries are removed from the output.

Yes this is where the decision happens which binaries are installed to $out/bin. I'd be willing to review a PR that updates inheritCargoArtifactsHook to accept doInstallProductionBinaries (defaults to true if not specified) and doInstallTestBinaries (defaults to false if not specified) and internally updates the jq filters accordingly

I think this should give callers the option to opt-into the behavior you are asking for

[MiniaczQ]

Wouldn't it be easier to manage and extend this if JQ pipeline was fully assembled in nix?

[ipetkov]

Wouldn't it be easier to manage and extend this if JQ pipeline was fully assembled in nix?

I don't think so, no. The current logic is written as a shell hook, so trying to assemble the pipeline during evaluation time means:

1. We preclude any build setups where the control happens at build (run)time and would force the settings to be set at Nix evaluation time
2. It would make including the hook a lot more complicated as we'd have to pass those arguments to it