Add composefs-ostree and some basic CLI tools

[alexlarsson]

Based on ideas from #141

This is an initial version of ostree support. This allows pulling
from local and remote ostree repos, which will create a set of
regular file content objects, as well as a blob containing all the
remaining ostree objects. From the blob we can create an image.

When pulling a commit, a base blob (i.e. "the previous version" can be
specified. Any objects in that base blob will not be downloaded. If a
name is given for the pulled commit, then pre-existing blobs with the
same name will automatically be used as a base blob.

This is an initial version and there are several things missing:

• Pull operations are completely serial
• There is no support for ostree summary files
• There is no support for ostree delta files
• There is no caching of local file availability (other than base blob)
• Local ostree repos only support archive mode

[allisonkarlitskaya]

I love this! Thanks for working on it!

I made some comments on the first round of commits. Feel free to adjust those and PR them separately: we can merge those now without further discussion.

The blobs thing is going to need a call.

I didn't review the crate addition in any detail at all. That's probably also going to need a call :)

[alexlarsson]

Hmmm, thinking more about this. We probably want a "content type" magic thing in the splitstream header as well, so we can error out if the wrapped thing is of the wrong type.

[alexlarsson]

Ok. Reworked this to use splitstreams for object maps and commits. And, by using an object mapping to find the object map we make the content of the splitstream for the commit be just the commit data, and thus the sha256 of that splitstream matches the ostree commit id.

[alexlarsson]

@allisonkarlitskaya There is still lots to do here. But have a look at this approach and see what you think.

[alexlarsson]

Added some further changes. We now validate all objects when pulling and all non-file objects when creating images. Its hard to efficiently validate file objects during create-image though, we would like to avoid re-reading the external object files to compute the sha256.

Remaining things to do:

• Stream larger objects into repo
• Support summaries and summary branches for remote repos
• Support deltas when remote pulling
• Parallelize downloads of objects
• Report pull progress in some sane way
• Use some kind of local cache for available objects other than just those from "previous version"
• Handle GPG validation of commit objects

[alexlarsson]

I started working on the delta support, but it failed because of an issue in gvariant-rs.

[allisonkarlitskaya]

It occurs to me that it might be interesting not to sort the table of fs-verity references, and it might also be interesting to permit duplicate items.

On the topic of deferring writing of objects to a background thread, this would allow us to write "external object #123" based on a sequential index to the splitstream without actually knowing the hash value yet, and then fill in the actual values in the header at the end when we're writing: it helps there that the fs-verity references aren't compressed and therefore not part of the stream...

[cgwalters]

It seems like we should get in the splitstream changes in 0f6d69e at least sooner rather than later? Can you file a separate PR?

[alexlarsson]

I rebased the code I had here on top of the splitstream changes that was merged in main. I removed all the code to give ostree commits some kind of named ref for now, as we need to figure out exactly how that will look. But, it has the basic minimal support to pull from a (remote or local) ostree repo into a ostree-commit-* splitstream object, and you can specify another such splitstream as a base name, and it will only download objects not in that version. (Which eventually we want to automatically pass what was the previous commit of the named ostree ref).

It also supports converting ostree commits to images, which can then be mounted.

There are tons of things that could be added on top of this:

• Named refs support
• Support summaries for remote repos
• Use of static deltas when downloading from http
• Some kind of object file map caching between commits other than the specified base name
• Parallel pulling with progress
• gpg validation
• Handle more local repo modes (currently only archive and bare-user-only

To test this you can run for example:

$ cfsctl --repo repo ostree pull-local /var/lib/flatpak/repo remotes/flathub/app/org.gnome.gedit/x86_64/stable
$ cfsctl --repo repo pull --base-name ostree-commit-306eed5546b7af406377e4947a118bd942729d9b380099fb191728654e2c925b https://dl.flathub.org/repo/ app/org.gnome.gedit/aarch64/stable
$ cfsctl --repo repo ostree create-image --image-name gedit ostree-commit-306eed5546b7af406377e4947a118bd942729d9b380099fb191728654e2c925b
$ sudo cfsctl --repo repo mount refs/gedit mnt

[alexlarsson]

The CI failures seem unrelated to this change. Weird.