Download proxied content to disk

[stv0g]

I would like to investigate to which extend it is possible to use nix-buildproxy for building a website which relies on astro-embed (see also delucis/astro-embed#215)

astro-embed fetches remote content such as tweeks, website previews, ... in order to embed them into the build output.

As these remote contents are usually relatively small, I would prefer to add them into my local Git repo.
This would also improve reproducibility as it might happen that certain remote resources (e.g. a Tweet) vanishes and can not be refetched.

Do you think such a use case would fit into the scope of nix-buildproxy?

[stv0g]

On a second look, I have realized that this use case is close to be already supported with the current inventory format.

We would simply add a path interpolation:

@@ -1,9 +1,6 @@
 { fetchurl }: [
   {
     url = "https://raw.githubusercontent.com/NixOS/nixpkgs/ba563a6ec1cd6b3b82ecb7787f9ea2cb4b536a1e/pkgs/by-name/he/hello/package.nix";
-    file = fetchurl {
-      url = "https://raw.githubusercontent.com/NixOS/nixpkgs/ba563a6ec1cd6b3b82ecb7787f9ea2cb4b536a1e/pkgs/by-name/he/hello/package.nix";
-      hash = "sha256-dFkeANLBJW1FWfL0d8ciS4siWP7B4z0vGsj9revgWGw=";
-    };
+    file = "${./inventory/some-file}";
   }
 ]
\ No newline at end of file

The missing piece here, would be to extend the buildproxy-capture tool with some command line parameter, which instructs it to download the proxied requests to a local file.

Ideally, we would make this configurable on a per-request bases using some sort of predicate.
E.g. based on mime/type, response body size, or so..

[polygon]

I like that idea a lot.

Not sure if I like extending the functionality of the existing buildproxy-capture. Maybe we could have a new tool, e.g., buildproxy-vendor. I'd imagine that it takes the inventory (which, if successfully built, will have all the artifacts in the Nix store). It could then copy the files from the Nix store to the repository and create a new inventory with local links as you described above.

[stv0g]

I like that idea too. Vendoring would also a good term to describe it. This we spring command could also give the user a bit more fine grained control over what should be vendored.

I am just wondering if there is a possibility to simplify the workflow a bit. E.g. by perform the proxying fully in the build sandbox and to build the whole inventory as a fixed-output derivation.

Vendoring could then just take this FOD and copy it to the repo/flake.