Incorrect include/exclude behavior with git dependencies (doesn't match `cargo package`)

[vikanezrimaya]

Describe the bug

Package with a git dependency that has a really complicated include section in Cargo.toml doesn't build, because ripgrep excludes files that cargo package would leave included. (There are two conflicting rules, which Cargo and ripgrep seem to interpret differently; cargo allows the more specific include rule to take precedence over the more vague exclusion rule, while ripgrep excludes it based on the vague wildcard rule).

Reproduction

$ nix build -L github:vikanezrimaya/crane-git-dep-include-bug

Additional notes
Adding --debug to the rg invocation in lib/downloadCargoPackageFromGit.nix confirmed the specific rg interpretation described above, while running cargo package in the git checkout of the aforementioned dependency and inspecting the .crate files confirmed that Cargo's behavior matches my expectations.

Perhaps it would be better to simply run cargo package in the crate checkout and then unpack the crates into $out instead of emulating Cargo's behavior using ripgrep?

[vikanezrimaya]

Tried replacing crateFiles definition with

cargo package --frozen -l -p "$crateName" | grep -v -e "^Cargo.toml.orig" $(if [[ ! -f Cargo.lock ]]; then echo "-e^Cargo.lock"; fi) | sort)"

The Cargo.lock exclusion is needed because, presumably, cargo package repackages the workspace Cargo.lock into every crate, and we apparently don't do that. So we ignore it if it doesn't exist. Same with Cargo.toml.orig that doesn't exist because we don't pre-process Cargo.toml.

Unfortunately this fails on a different crate in my current project I'm building; cargo for some reason wants to read the crates.io index (why doesn't it want to on the package I was originally having trouble with? no idea).

[ipetkov]

Hi @vikanezrimaya thanks for the report! I couldn't quite remember if there was a reason cargo package wasn't used, though I also just tried it and it seems that cargo insists on having an index present and errors out due to no network access.

I also noticed that cargo package --list exists (for printing out what files would be packaged) but that too suffers from the same problem...

I'm definitely open to improving this (or switching to using cargo to the degree that we can) but sadly this doesn't seem possible right now... Still, open to any PRs to improve this or fix any edge cases!

[vikanezrimaya]

@ipetkov Could we, perhaps, attempt to extract the relevant code from Cargo itself? That way we could make sure we aren't doing any network access, or, at least, understand what Cargo does and why does it need the registry index for this operation. I think it might be worth it for me to look into this. This could also be a good opportunity to tinker with Cargo a little…

[ipetkov]

If you have an appetite for working with cargo's internals I'd suggest exploring a way to extract or enhance the existing functionality to allow offline "pre-packaging"! Having this be maintained upstream would make it a lot less brittle than trying to maintain it externally