Skip to content

Replace ceph by cephadm#364

Open
dupremathieu wants to merge 24 commits intoscarthgapfrom
replace-ceph-by-cephadm
Open

Replace ceph by cephadm#364
dupremathieu wants to merge 24 commits intoscarthgapfrom
replace-ceph-by-cephadm

Conversation

@dupremathieu
Copy link
Member

Summary

This PR migrates the SEAPATH cluster storage management from bare Ceph daemon
packages to cephadm, the official Ceph orchestration tool. Cephadm manages
Ceph daemons as containers rather than native system services. Since cephadm
relies on Podman (rather than Docker) to run those containers, this
migration also implies replacing Docker with Podman as the container runtime
dependency. Finally, reworking the livemigration user—now renamed
libvirtadmin to reflect its broader role—was necessary to keep user
management consistent throughout the cluster.

All other commits in this PR address bugs and regressions discovered during
the migration work.


Main changes

cephadm integration

  • recipes-extended/ceph: create the cephadm recipe – Introduces a
    dedicated Yocto recipe that fetches the upstream cephadm Python script and
    installs it on the target image.
  • distro: seapath-common: set cephadm as preferred cephadm provider
    Declares the new cephadm recipe as the preferred provider for the
    cephadm runtime package, so it supersedes the copy bundled inside the
    monolithic ceph package.
  • images/seapath-host-common-ha.inc: install cephadm package – Adds
    the cephadm package to the HA host image so the orchestrator is available
    at first boot.
  • cephadm: add lvm2 and podman dependencies – cephadm requires both
    lvm2 (OSD provisioning) and podman (daemon containers) at runtime;
    these are now declared as explicit RDEPENDS.

Podman replacing Docker

  • The kernel configuration fragment docker.cfg is retained for the
    container-related kernel options that Podman also needs.
  • Docker packages (docker-ce, docker-ce-contrib) are removed from the
    host image recipes and replaced where appropriate by podman and related
    packages.

User management rework

  • system-config-cluster: rename user to libvirtadmin – The
    livemigration user/group is renamed to libvirtadmin because the account
    is now also used for VM console access across the cluster, not only for live
    migration. The rename is applied consistently in:
    • conf/distro/include/group.gid
    • conf/distro/include/passwd.uid
    • cukinia security tests (groups.conf, passwd.conf)
  • python3-vm-manager: bump to the last version – Picks up the upstream
    change that uses libvirtadmin as the default user for the vm-mgr console
    command, and removes the now-unnecessary pyproject.toml licence field
    patch.

Ceph packaging refactor (prerequisite work)

A series of commits restructured the Ceph 18 Yocto packaging to make the
cephadm migration feasible and to align with Debian's package split:

  • recipes-extended/ceph: add ceph.inc and include it in ceph_16.bbappend
    Extracts common packaging logic into a shared include file.
  • recipes-extended/ceph: import ceph 18.2.7 from meta-virtualization
    Brings in the upstream Ceph 18.2.7 recipe as a baseline.
  • recipes-extended/ceph: add ceph 18 bbappend – Applies SEAPATH-specific
    customisations on top of the base recipe (cross-compilation patches, file
    installations, sysctl configuration, sudoers, etc.).
  • ceph: refactor packaging into smaller packages – Splits the monolithic
    ceph package into fine-grained sub-packages (ceph-common, ceph-base,
    ceph-mon, ceph-osd, ceph-mds, ceph-mgr, …) matching Debian's
    structure, enabling minimal installations.
  • ceph: remove circular dependencies between ceph packages – Fixes a
    packaging deadlock where each sub-package depended on the main ceph package
    and vice-versa. Introduces CEPH_BASE_PACKAGE to control which package owns
    the ceph system user.
  • ceph: move packages definitions back to ceph 16 bbappend – Consolidates
    package definitions in the bbappend to avoid duplication between the .bb
    and .bbappend files.
  • distro: seapath-common: update ceph version to 18.2.7 – Promotes
    Ceph 18.2.7 as the default cluster storage version.
  • ceph: patch updates – Ports the Python binding cross-compilation patch
    from Ceph 16 to Ceph 18 and imports relevant patches from Debian.
  • ceph: update patch Upstream-Status – Marks OE-specific patches
    correctly to avoid spurious upstream submission warnings.
  • ceph: disable export BUILD_DOC in ceph_18.2.7.bb – Disables the
    BUILD_DOC export that was enabling the Ceph library mock-up used for
    documentation generation, which caused build issues.

Testing

The cukinia test suite has been updated throughout this PR to reflect the new
user/group names and the new cephadm-based deployment model:

  • SEAPATH_CLUSTER_USERS now includes libvirtadmin instead of
    livemigration.
  • Cluster-specific tests (ceph OSD path, cluster user presence) are guarded
    behind the seapath-clustering DISTRO_FEATURES flag so they do not run
    on non-clustering images.

To test cephadm instead of Ceph daemon on Yocto you do the following modification
in your inventory:

  • Remove: force_cephadm: true
  • Add the following global "vars":
cephadm_pullimages: true
force_cephadm: true
cephadm_network: "{{ public_network }}"

To avoid issues with vim and systemd status output not displaying
correctly due to missing terminfo files, we add the ncurses-terminfo
package to all SEAPATH images.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
This fragment enables the necessary kernel options to run Podman
containers.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Ceph package includes daemons. We not longer want them on seapath-host
images because cephadm will take care of deploying them when needed.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Add a patch to change the storage graphroot and rootless storage path to
point to a persistent volume mounted at /mnt/persistent.

We have to because we cannot use overlays on top of another overlay
filesystem, which is the case when using the default
/var/lib/containers/storage path on our systems.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Podman need / and /mnt/persistent to be rshared to work properly.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Recent versions of Podman use psst as the default network driver for
rootless containers, but it is not available in the scarthgath version
of meta-virtualization. This commit configures Podman to use slirp4netns
instead.

When we will update to a newer version of Yocto, we will be able to
remove this configuration and use the default network driver.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
We have changed the Podman user storage directory to be on a persistent
volume. Now we need to create this user storage directories there with
the right permissions.

This commit adds a systemd service that creates the user
directories on boot, and a script that does the actual work of creating
the directories and setting the permissions.

The new variable USER_CONTAINER_LIST can be set to a list of users for
which the directories should be created.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
USER_CONTAINER_LIST is a list of users that are allowed to run rootless
containers. This is needed for cephadm to be able to run cephadm
containers as non-root users, which is a requirement for running cephadm
in rootless mode.

We also add the admin user to the list of users that are allowed to run
rootless containers.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
SEAPATH cephadm role needs ceph-volume to be able to manage OSDs. This
commit adds it to the list of packages installed in the image.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
The ceph-volume package requires the python3-packaging module to be able
to run.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Create the lvm's tmpfiles.d configuration file and install it in the
lvm2 recipe. This is required by cephadm to work properly.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
The ceph user creation was made twice, once in ceph.inc and once in
ceph_16.2.15.bb, which caused a warning about USERADD_PACKAGES being
overridden. This commit removes the redundant definition from
ceph_16.2.15.bb and keeps it only in ceph.inc.

Also use USERADD_PACKAGES += instead of = in ceph.inc to avoid
overriding any previous value.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
The users configuration was a bit messy and completely broken. This
commit reworks the users configuration to be more consistent and easier
to maintain.

The sudo group (privileged) was configured to give all members ALL
permissions without a password. A user has to be a member of this group
to be able to run/execute sudo. This means if a user needs to be able to
run sudo, they need to be a member of the privileged group and have all
privileges without a password. This is total nonsense and a security
risk.

To fix that, this commit removes the sudo group (privileged) from the
sudoers configuration. This group is now only used to give users the
right to run sudo as it was specified.

In addition to that, this commit adds the possibility to specify for
each user if they need to have the NOPASSWD or EXEC tag in their sudoers
configuration, as it is possible for groups.

This is useful for special users like Ansible that need to be able to
run sudo without a password.

The commit also removes the need to add manually the sudo group
(privileged) to sudo users.

And finally, this commit also removes the ceph user configuration from
this file. The ceph user configuration is done in the ceph recipe; there
is no need to have it here as well.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
We have issue with cephadm if an unconfigured ceph.conf is present in
/etc/ceph/ceph.conf.

This file is not needed with cephadm and ceph-ansible, so we can safely
remove it from the image.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
Create the cephadm and containerized-ceph users and groups, and add
cephadm to sudoers. This is required to be able to run cephadm with the
right permissions to manage ceph daemons and containers.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
The sudo group is no longuer created by the user-config bbclass, we
have to create it in the sudo recipe, otherwise the sudoers file will
not be able to be installed.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
To avoid conflicts with our SEAPATH Ansible files, we need to update the
registries.conf file to use the new syntax.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
The Ansible sudo group was removed in favor of an Ansible user. This
change updates the policies.conf file to reflect this change.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
The hadening options were not ported to the 6.12 branch, and some of
them were missing. This commit restores the hardening options and
updates them to match the latest mainline configuration.

The hardening options are based on the Recommanded Settings of the
Kernel Self Protection Project (KSPP).

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
The configuration of the IOMMU test was wrong, the KSPP guidelines and
the R7 recommandation of the ANSSI BP028 require that the IOMMU be in
strict mode, but the test was checking for passthrough mode.

This commit updates the test to check for strict mode instead.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
We have to conduct latency tests before enabling the IOMMU in strict
mode. Keep the strict mode enable by default, but add a configuration
fragment to enable the passthrough mode.

Signed-off-by: Mathieu Dupré <mathieu.dupre@savoirfairelinux.com>
@eroussy
Copy link
Member

eroussy commented Feb 26, 2026

Why are you talking about libvirtadmin user. You don't do anything related to this user in this PR
The username change from livemigration to libvirtadmin was already done in 72e06e7
In fact, all the "User management rework" is from another PR. Remove it from the description and remove all mention of libvirtadmin

@eroussy
Copy link
Member

eroussy commented Feb 26, 2026

Ok to remove docker, but we should install the docker-podman compatibility tool, so that user can still use the "docker" command

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants