refactor(default-flatpaks): version 2 (nushell)

[xynydev]

This PR is a draft of the rewrite of default-flatpaks to support multiple repositories, pioneer Nushell-based modules, and rethink the config structure.

Current version implements only the build-time part. Plan is to write to /usr/share/bluebuild/default-flatpaks/configuration.json at build time and read from there in the post-boot parts.

The recipe format is still up for debate. I didn't end up using anything in #146 for now. The current approach is basically the same as my original suggestion, except that one module call can install Flatpaks in multiple scopes from multiple repositories. The terminology still feels off. Feel free to suggest improvements.

Also feel free to look at the source code to see if you mostly understand it and if it is more readable than the v1 source.

The syntax highlighting on GitHub might be a bit weird, though, because I had to make the file end in .sh due to our current system hardcoding that in. I'm not sure what we could do about that, maybe make it possible to omit the file extension entirely.

Testing

recipe:

modules:
  - type: containerfile
    snippets:
      - "COPY --from=ghcr.io/blue-build/nushell-image:0.98.0 /nu/nu /usr/bin/nu"
  - source: ghcr.io/blue-build/modules:nu-flatpaks
    type: default-flatpaks
    installations:
      - notify: true
        scope: user
        repo:
          title: Flathub (user)
        install:
          - io.gitlab.leesonwai.Sums
          - io.github.nokse22.minitext
      - notify: true
        scope: system
        install:
          - xyz.splashmapper.Splash
          - xyz.safeworlds.hiit

example output:

[fiftydinar]

Regarding code structure, it looks cleaner, simpler & I generally like it.

Only thing which clutters things very much is echo-ing in different colours. So message is harder to see in code.
Also, we use colours in bluebuild cli for start & end of the module, so maybe it's better to use message colours only when necessarry for better visualization (f.e. red colour for error message).

Is JSON as a config format necessarry? I find .yml format cleaner. As you said, nu can be used to read/modify YAML files.

Also printing the message for every checked FlatHub flatpak ID is too much imo, it should only print success message once & print error when ID is wrong.

Recipe format looks good to me, except maybe instead of installations, modifications can be used.

Thanks for starting this PR, I'll look into learning Nushell, so I can also contribute

[xynydev]

Regarding code structure, it looks cleaner, simpler & I generally like it.
...
Thanks for starting this PR, I'll look into learning Nushell, so I can also contribute

Nice!

Only thing which clutters things very much is echo-ing in different colours. So message is harder to see in code.
...
Also printing the message for every checked FlatHub flatpak ID is too much imo, it should only print success message once & print error when ID is wrong.

Ok, maybe I went a bit overboard with that. I'll fix that in the next commits.

Sure

Also, we use colours in bluebuild cli for start & end of the module, so maybe it's better to use message colours only when necessarry for better visualization (f.e. red colour for error message).

Do we? I haven't seen that.

Is JSON as a config format necessarry? I find .yml format cleaner. As you said, nu can be used to read/modify YAML files.

It's not a config format for humans, it won't be writeable as it will exist only in the /usr/share/ directory. I thought that JSON would be better as a serialization format for data written and read by computers only. We can change that though if you wish, it's not a big deal.

Recipe format looks good to me, except maybe instead of installations, modifications can be used.

Those "modifications" can't actually modify each other, though... I chose "installations", because it IMO reflects that each entry means one set of Flatpaks installed from a specific repo to a specific scope. I guess it feels redundant with the install: key though. We might consider renaming that to applications: or something.

[fiftydinar]

Also, we use colours in bluebuild cli for start & end of the module, so maybe it's better to use message colours only when necessarry for better visualization (f.e. red colour for error message).

Do we? I haven't seen that.

We can change that though if you wish, it's not a big deal.

It's easier to read for users who want to know what's modified in the image, hence the reason I insist on it.
But it's not a big deal.

Those "modifications" can't actually modify each other, though... I chose "installations", because it IMO reflects that each entry means one set of Flatpaks installed from a specific repo to a specific scope. I guess it feels redundant with the install: key though. We might consider renaming that to applications: or something.

default-configuration:?

[xynydev]

default-configuration:?

I don't really like that either, because it's two-parted and does not feel like it makes sense for a list of separate configurations. Maybe configurations: would make sense?

[fiftydinar]

Maybe configurations: would make sense?

Looks good to me

[fiftydinar]

Looking at flatpak repo, I see that it got support for specifying system repos in /usr/share/flatpak/remotes.d/.

I think that we can specify repos there, with disadvantage that it can't be removed by local-user (or advantage, depending on how you look at it). In that case, /etc/flatpak/remotes.d/ can be used.

Shipping disabled repos OOTB is not supported though, so Fedora does it in a different way (just saying if wondering why Fedora doesn't utilize remotes.d).

This would need to be tested though.

[xynydev]

@fiftydinar

Latest commit contains initial implementation of the post-boot parts. There might be some overlooked functionality. You should look through especially the TODOs in the system setup script, if you want to implement some part of this PR.

[xynydev]

I tried to search for information about flatpak/remotes.d/ but couldn't find anything substantial. We or you shall look into it again later.

[fiftydinar]

Not all points addressed (will think later about notify-send), but here's my initial feedback

[fiftydinar]

I merged the changes I mentioned in review to default-flatpaks-v1. They are even more improved.

So please look there for my updated suggestions.

[fiftydinar]

@xynydev
About making it easier for users to disable the module, I agree. That can be maybe implemented in default-flatpaks-v1 even.
But, what is the definition of module disablement by the user to you?
Disabling systemd services from running? Or additionally, erasing any modifications that the module did (installation of flatpaks)?

About not implementing flatpak removal & by that, user-config functionality for default-flatpaks-v2, I have an opinion that this should be ideally implemented when we have the solution for issue #188, but since we don't have a solution for that, I think that I agree with you on this one when looking from general angle, even if I subjectively find this feature useful.

[xynydev]

About making it easier for users to disable the module, I agree. That can be maybe implemented in default-flatpaks-v1 even.
But, what is the definition of module disablement by the user to you?
Disabling systemd services from running? Or additionally, erasing any modifications that the module did (installation of flatpaks)?

I think my definition is just disabling the systemd services. But it may be documented that a user could cat the file with the Flatpak configurations to see if there's any there that they want to undo manually. We could also provide scripts in libexec (that downstreams could hook up to ujust or something if they want to) that would undo all the changes automatically I guess.

Ideally the same scripts that are used as systemd services could also be run manually from the terminal. Those scripts would also be able to confirm the changes (including deletions, if we implement that) before doing them. There's possibilities there, one is allowing image maintainers to set how deletions are done and what changes need user confirmation, how that can be given, etc.

[xynydev]

will think later about notify-send

If there's no clean way, we could just say that it's unsupported. It seems you solved this in v1 somehow, though.

[fiftydinar]

If there's no clean way, we could just say that it's unsupported. It seems you solved this in v1 somehow, though.

Well, you can implement it the way I did in v1, or if you think it's too hackable, drop it. I think it is the only way to reliably show notifications.

[fiftydinar]

Our current setup is to only put what's necessary in /usr/share/bluebuild/${module}/. And that's configuration.

Currently, we additionally put binaries & systemd services + timers there. I think that we should copy them directly to their appropriate locations instead of having them in /usr/share/bluebuild/${module}/ location.

What do you think?

[xynydev]

re: 993d5c6

I realized that the fancy error message would never show, and the user would get caught up in rather frustrating and confusing schema validation errors instead, so I changed the schema such that configurations with type: default-flatpaks using the old schema will still validate properly. This will allow the module script to be run and output the nicer error message.

The downside is potentially confusing in-editor validation, but I think it should be fine, since the new schema still does not allow mixing and matching new and old schemas together. After a transitory period, we may of course revert this change.

[xynydev]

re: ffd8d5a

I think this is super cool and useful. I already tested it locally, and it works well. Still gotta decide if we need to allow disabling the installation of this tool, in case someone doesn't want it. I guess that decision could wait until someone complains as well, it's a pretty quick change...

[xynydev]

This is nearing ready now (there wasn't really a ton of work left to begin with, I just haven't gotten to it). So, I'll mark as ready for review. Still got some testing to do (which I'm doing right now), and probably some amount of discussion as well. Lastly, I/we need to write the announcement post. That is something for another day, though.

[fiftydinar]

re: ffd8d5a

I think this is super cool and useful. I already tested it locally, and it works well. Still gotta decide if we need to allow disabling the installation of this tool, in case someone doesn't want it. I guess that decision could wait until someone complains as well, it's a pretty quick change...

I also think it's cool & I would leave that.

It may also be automated by image-maintainer if he desires so.

[fiftydinar]

This is nearing ready now (there wasn't really a ton of work left to begin with, I just haven't gotten to it). So, I'll mark as ready for review. Still got some testing to do (which I'm doing right now), and probably some amount of discussion as well. Lastly, I/we need to write the announcement post. That is something for another day, though.

Looks generally good (as far as I can understand nushell, which is still strange to me compared to my natural POSIX world)

[xynydev]

Testing left to do

• Try rebasing a clean VM with both system and user Flatpaks to configured to install, some Flatpaks from the Fedora remote as well, and a third Flatpak remote too if we can find one
• Try all modules with the latest Nushell version and upgrade

After that, just announcement writing and merging... Woohoo!

[xynydev]

Try rebasing a clean VM with both system and user Flatpaks to configured to install, some Flatpaks from the Fedora remote as well, and a third Flatpak remote too if we can find one

Doing this today. Test configuration used is below. User Flatpak setup concludes in ~7min. System Flatpak setup has some problems right now.

type: default-flatpaks
configurations:
      - scope: user
        install:
          - com.brave.Browser
          - com.github.tchx84.Flatseal
          - page.kramo.Cartridges
          - org.musicbrainz.Picard
          - org.speedcrunch.SpeedCrunch
      - scope: user
        repo:
          title: Fedora Flatpaks
          name: fedora
          url: oci+https://registry.fedoraproject.org
        install:
          - org.mozilla.firefox
          - org.blender.Blender
      - scope: user
        repo:
          title: Flathub Beta
          name: flathub-beta
          url: https://flathub.org/beta-repo/flathub-beta.flatpakrepo
        install:
          - org.godotengine.Godot
      - scope: user
        repo:
          title: GNOME Nightly
          name: gnome-nightly
          url: https://nightly.gnome.org/gnome-nightly.flatpakrepo
        install:
          - org.gnome.Epiphany.Devel
      - scope: system
        install:
          - io.itch.itch
          - page.codeberg.zynequ.Kapitano

EDIT: Ok, I think the issues I encountered should be just about fixed now. I'll mark the testing as done. If @fiftydinar wants to do Fedora flatpak related changes, those are very welcome, because I'm not 100% certain about the implementation right now, but I think we should currently be in a functional state regarding that aswell.

[fiftydinar]

If @fiftydinar wants to do Fedora flatpak related changes, those are very welcome, because I'm not 100% certain about the implementation right now, but I think we should currently be in a functional state regarding that aswell.

Let's keep it as it is, in case they become useful for something. Enabling repos immediatelly after adding the repo is good enough.

[xynydev]

Announcement written. blue-build/website#93

Waiting for reviews from others. Ready for merge whenever.