Bugfix: new packages should be downloaded in the base layer

[leandrosansilva]

The use case for it is that the user might want to use add-packages-to-pool where the packages come from extra (non-ubuntu) repositories.

In my case, I setup the repositories in the base layer and noticed they were not used in a further add-package-to-pool action, as add-package-to-pool was being executed in a different layer.

The Ubuntu Server version I am using is 22.04.

I've been using this change in production for some time now and it works well for me.

I still perform some extra steps to add extra repositories, but I haven't gotten yet a good interface for it on livefs-editor, so that will keep in a further pull request :-)

[leandrosansilva]

Just pinging to check if anyone has time to review this change :-)

[mwhudson]

I don't particularly like this as you do it here, as it will always repack the base layer (with updated apt lists, if nothing else). But what I guess --add-package-to-pool could do is see if the base layer has already been mounted for editing?

(BTW, what are you doing in general? saying you use this script "in production" is mildly alarming. I'd rather we had easier ways for people to build ISOs that resembled the release ISOs rather than this pile of hacks)

[jasimioni]

Hello,

I'm also interested in this feature, and I'm using a very similar change to accomplish the same.

@mwhudson, when you say the --add-package-to-pool would check for the base layer to be mounted, do you mean the new/ubuntu-server-minimal, and if it is use the edit_squash, but if not, use the regular behavior? If that is the case, would a snippet like this be better?

    name = get_squash_names(ctxt)[0]
    reltarget = f'new/{name}'
    target = ctxt.p(reltarget)
    if os.path.exists(target):
        dir = ctxt.edit_squashfs(get_squash_names(ctxt)[0])
    else:
        fs = ctxt.mount_squash(get_squash_names(ctxt)[0])
        overlay = ctxt.add_overlay(fs, ctxt.tmpdir())
        dir = overlay.p()

    cache = cache_for_dir(ctxt, dir)

[mwhudson]

Yes, something like that, although that code seems a bit of an abstraction violation (not that this is the cleanest codebase ever!)

[jasimioni]

I also thought about that, but there are some code snippets in actions.py that use a similar trick, referencing directly the folders and using ctxp.p(), so I ended up suggesting this way.

I can add a method to context.py to do this check to remove the logic from actions.

[leandrosansilva]

I don't particularly like this as you do it here, as it will always repack the base layer (with updated apt lists, if nothing else). But what I guess --add-package-to-pool could do is see if the base layer has already been mounted for editing?

(BTW, what are you doing in general? saying you use this script "in production" is mildly alarming. I'd rather we had easier ways for people to build ISOs that resembled the release ISOs rather than this pile of hacks)

Hi @mwhudson, glad to know you're alarmed with livefs-editor being used in production.... haha

Explaining what I mean by "used in production", this is my current use case:

At a client, several pieces software, each living in their own gitlab repository are built and automatically packaged on their own releasing cycle. Some of the custom software unfortunately depend on wine32.

Every night, though, an automated gitlab pipeline fetches all the deb packages from different repositories and generates an ISO image.

In addition to the default ubuntu repositories (amd64 and i386) and our custom deb packages, we also add some 3rd party repositories, which is the reason for the current PR, as livefs-editor by default does not play with 3rd party repos.

When we want to generate a new "bundle", such nightly image is manually tested and promoted to a release to be used by operations team.

The product is not really a "server" in the modern sense (deployed in the cloud or providing services over the internet), but "embedded" systems running on physically isolated locations, mostly offline or with limited connectivity.

For instance, we use chrony as NTP implementation as it can synchronize the internal clock via GPS, to prevent it of drifting too much when there is no connectivity with the Internet.

The installation is totally automated via subiquity/autoinstall.yml with lots of customization, for instance, for handling installing stuff running on wine32.

This is probably not the usecase originally thought for livefs-editor, but it this setup has been working well for the past months, and it randomly broke recently, being fixed by the fix in the current PR.

I hope that helps you understand better my use case :-)