Recipe versioning

[sage-etcher]

I'm unsure if this should be posted under Recipes, Compile, or Scripts. if this is the wrong place for it feel free to move it or lmk.

Feature Request

Having an internal versioning system for recipes, as a way to check if the recipe you used has had any changes since you compiled the program.

Potentially through logging the recipe's source and commit to a Resources file.

Issue

Say I install Bash 5.1 from Recipe, and then a few days later an issue is found+fixed in the recipe. Currently I'd have no way of knowing (w/o manual checking of dates) if I used the version before or after the Recipe's change.

This issue is found first hand with the installed version of Bzip2 1.0.8 provided in Gobo017, having used an outdated Recipe file, leading to some programs failing to find it's shared library.

Proposed Method

Having Compile (or an adjacent script) log some information about the Recipe, for exanple using git log [FOLDER_PATH] |head -1 > $target/Resources/RecipeCommit to get the commit.

Then be able to check it against the current version in Recipes as follows:

local name="$1"     # program name
local version="$2"  # program version

local recipesfolder="/Data/Compile/Recipes/${name}/${version}"
local installedfolder="/Programs/${name}/${version}"

local recipescommit=$(git log "${recipesfolder}" |head -1)
local installedcommit=$(cat "${installedfolder}/Resources/RecipeCommit")
if [[ "${recipescommit}" == "${installedcommit}" ]]
then
   echo "up to date"
else
   echo "out of date"
fi

Ideally, a bit more than just the commit would be logged, potentially the git repository/fork, branch, and commit; just something concrete, fast to generate, and easy to compare.

[mwh]

Recipes used to have -r1, -r2 suffixes for exactly that, but they were dropped I think in the Git migration. That does mean that the tooling likely still has full support for them and it would be possible to revive those (though it's not clear whether that's the right way to go about things still).

[nuc1eon]

Personally I'd be in favor of reviving the -r1, -r2 approach. And yes I agree that this would be a very useful thing to have.

[nuc1eon]

Recipes used to have -r1, -r2 suffixes for exactly that, but they were dropped I think in the Git migration. That does mean that the tooling likely still has full support for them and it would be possible to revive those (though it's not clear whether that's the right way to go about things still).

@mwh Would you mind detailing how this was applied in practice?

For instance a program "Bash 5.0.0", had that been called Bash 5.0.0-r1 from the get-go in the recipe store? Or had the -r1, -r2, ... suffixes been added only in case the original recipe was altered?

[mwh]

All published recipes started at -r1, so the initial packed recipe file on the recipe store would have been Bash--5.0.0-r1--recipe.tar.bz2. I believe the SVN tree had separate subtrees for the primary recipe and revisions and the maintenance tooling made the copies. Once things moved to git, the whole recipe store is always fetched and there is only the primary tree.

[nuc1eon]

I'd dislike the additional clutter of appending a -r1 to every recipe version, also due to the fact that amendments to existing recipes are more of an exception rather than a rule.

Maybe we could assume an unwritten -r0 on every initial recipe, and append -r1, -r2, ..., -rn to subsequent recipe changes.

Thoughts @sage-etcher ?

[sage-etcher]

Either approach is equally fine by me, I believe both should be supported by the tooling already. While debugging recipes I use an implicit -r0 and haven't run into any issues doing such, though at the same time, I don't feel -r1 would be too cluttered.

A few concerns I have with that approach:

• Size, as more alterations are made the repo size will grow with packages that are not intended to be used.
• Blame, git blame is a real helpful tool for finding where different changes originated from, if every alteration is a new copy/paste, the blame wont be able to source changes any longer. This is already present when sourcing which version different patches originated from.
• Why would we implement a custom version control when a preexisting one is built into git? The -rn solution seems to break a good bit of the git version control benefits? Feels odd implementing/enforcing a secondary version control system on top of the preexisting system.

I do like the human readable nature of the -rn solution though! A potential fix for all 3 concerns would be, renaming the recipe's version with each revision instead of copy/paste. This should allow git blame to do it's thing, and fix the size/unused recipe issues. If we need to roll back to a previous revision then we have git, that's git's job. Gobo's tooling should still be able to parse that a new revision is out, without the clutter of keeping all previous revisions constantly available.

[mwh]

I have to imagine that having an extra bit of the version in some published recipes that needs to be chopped off for the filesystem directory and for dependency handling, and not in others, is going to be pretty fragile. However, usually LocalRecipes didn't have revision tags in it, only the main fetched-as-tarball recipes did, although I suppose when working on recipes it was expected that you might be pushing things.

Removing old versions in general is a pretty significant change to the model when installing multiple parallel versions is traditionally one of the selling points. The tooling could be updated to handle that at the same time, but it seems like it'd require a lot of specialised git handling and embracing it as a database to get the same effect. That might be worthwhile if the outcome is desirable, but it is quite a lot of work.

It's not obvious that one big git repository of recipes is the theoretically-right way to go about things other than for (very real) offloading of hosting reasons. This whole issue is one of the limitations it brings.

[nuc1eon]

@sage-etcher I understand your concerns. Not being able to track changes properly across versions has already been a thing.

And adding the concept of recipe revisions would amplify this limitation quite drastically, because it would be quite hard to properly review and compare a new PR against the previous state (and makes it harder to track changes over time), I confess.

Though, I dislike the fact that we'd have to adapt to git as a versioning system, rather than the versioning system be fit for our needs. It almost appears that git is not the appropriate tool to handle our recipe store in its current form.

Size, as more alterations are made the repo size will grow with packages that are not intended to be used.

We could instead keep our recipes in separate folders by GoboLinux release version. Eg Recipes/017/, Recipes/017.01. With subfolders: Core, Extra, Archive. Core keeps all packages on from a typical GoboLinux ISO, while Extra keeps additional maintained (or known working) recipes. Archive would keep most of the current untested kitchen sink (which I guesstimate is currently at least 90% of our recipes frankly).

So that would do away with 90% of the clutter, and we can further thin down versions we keep available.

I agree that being able to keep multiple versions of a program around in parallel is a major selling point, however there's no point in keeping dozens of Linux 2.x.x kernel recipes around.

So once we clean the that out to a manageable level, our current system should already feel a lot more comfy in day-to-day life:)

Why would we implement a custom version control when a preexisting one is built into git? The -rn solution seems to break a good bit of the git version control benefits? Feels odd implementing/enforcing a secondary version control system on top of the preexisting system.

We will have to make a decision whether: we switch to an ArchLinux-like model (that keeps every package in a seperate repo), stick to our current system, or come up with a custom solution...

1. The "ArchLinux" model (for reference) for our recipes would have several implications, some of which @mwh already mentioned. Also it would be a bigger effort than appears on first sight (my own estimation).

2. If we stick to our current system, we'd have to get some better tooling to properly review and track changes across our recipe store (in the long run). Could it be that GoboLinux used to have exactly that with https://github.com/gobolinux/RecipeViewer and https://github.com/gobolinux/ReviewPanel ? I've never witnessed these tools in action.

3. A custom solution could look like this, which is a mix of the above:

• We retain our current directory layout
• Each Recipe is it's own git repo (but stored in one big repo)
• Each version exists in its own branch (for instance branch 5.0.0 contains one folder 5.0.0/), as well as one of the branches stable, testing, or archive (which each contain a select conglomerate of version folders)
• PRs are merged to their corresponding version branch
  • A commit message like: "Fuse 2.9.8-r3: Fix build" would set a git tag with "2.9.8-r3, 3.9.8-latest" on that commit
  • Branches are the upstream version numbers; tags are our downstream recipe revisions (and always include an -rn suffix)
  • The version branch is merged subsequently into the stable, testing, or archive branch (including revision git tag)
  • This process can be automated via a GitHub runner

Following that logic we could do the same for the entire repo (suggested layout):

Recipes [overarchig repo]
├── 017 [git branch: 017]
│   ├── Archive
│   ├── Core
│   │   └── Fuse [git submodule]
│   │       [git branch: stable (other branches: testing, archive, + one branch for each version)]
│   │       └── 2.9.8 [git tags: 2.9.8-r3, 2.9.8-latest]
│   │           [git branch: 2.9.8]
│   │           └── [git tag: 2.9.8-r2]
│   │           └── [git tag: 2.9.8-r1]
│   │       └── 3.0.0 [git tags: 3.0.0-r0, 3.0.0-latest]
│   │       └── ...
│   │   └── Gawk [git submodule]
│   │       └── ...
│   └── Extra
├── 017.01
│   ├── Archive
│   ├── Core
│   └── Extra
└── 020
    ├── Archive
    ├── Core
    └── Extra

Pros:

• The user has a clear view on relevant recipes in the stable recipe tree and gets easy access to others
• The user doesn't need to clutter their local recipe store with archaic recipes (but can choose to do so via testing, archive branches)
• Having every recipe git branched and tagged allows us to easily compare changes across versions and revisions: git diff 5.0.0...5.0.1, git diff 5.0.1-r1...5.0.1-r7 (this should also be possible on GitHub)
• PRs to existing versions will show a nice diff via GitHub's UI and git blame will work, too
• The branches stable, testing and archive could let us manage recipes quite nicely in the long run
• Git handles branching efficiently - we don't have to worry about wasting space with duplicated content
• No need to change our recipe's folder structure; remains coherent with Programs/ (apart from being split into Core/, Extra/ and Archive/ folders)
• Good occasion to give our tool ContributeRecipe a new shine!

Cons:

• it's work :P
• maybe you have some?

EDIT:

• A recipe store of thousands of submodules is more complex than one big Monorepo; also requires some automation to keep branches in sync and tooling to make sure that changes are commited at the right place

I'd be curious for your input on the above!