Issues with Nydus Snapshotter Installation in RKE2/K3s Environments

[sandlbn]

Summary

When using the Confidential Containers operator with RKE2 or K3s Kubernetes distributions, the pre-install daemon consistently attempts to install and configure Nydus Snapshotter even when explicitly setting INSTALL_NYDUS_SNAPSHOTTER: "false" in the CcRuntime resource. This causes the pre-install process to fail when it tries to restart the non-existent or masked containerd service, as RKE2/K3s embed and manage containerd differently than standard Kubernetes installations.

Environment

Kubernetes distribution: RKE2 v1.24.17+rke2r1
Confidential Containers operator version: (latest from main branch)
Single-node cluster with control-plane and worker roles on the same node

Steps to Reproduce

• Install RKE2 on a server
• Deploy the Confidential Containers operator
• Create a CcRuntime resource with INSTALL_NYDUS_SNAPSHOTTER: "false" explicitly set

apiVersion: confidentialcontainers.org/v1beta1
kind: CcRuntime
metadata:
  name: ccruntime
spec:
  runtimeName: kata
  ccNodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""
  config:
    installType: bundle
    payloadImage: quay.io/kata-containers/kata-deploy:stable
    environmentVariables:
      - name: "CONFIGURE_CC"
        value: "yes"
      - name: "DEBUG"
        value: "true"
      - name: "INSTALL_COCO_CONTAINERD"
        value: "false"
      - name: "INSTALL_OFFICIAL_CONTAINERD"
        value: "false"
      - name: "INSTALL_VFIO_GPU_CONTAINERD" 
        value: "false"
      - name: "INSTALL_NYDUS_SNAPSHOTTER"
        value: "false"
    runtimeClasses: 
      - name: "kata"
        snapshotter: ""
        pulltype: ""

Despite explicitly setting INSTALL_NYDUS_SNAPSHOTTER: "false", the pre-install daemon still attempts to install and configure Nydus Snapshotter as evidenced by the logs:

INSTALL_COCO_CONTAINERD: false
INSTALL_OFFICIAL_CONTAINERD: false
INSTALL_VFIO_GPU_CONTAINERD: false
INSTALL_NYDUS_SNAPSHOTTER: true  # Note: This is set to true despite config setting false
Copying nydus-snapshotter artifacts onto host
Unit /etc/systemd/system/nydus-snapshotter.service is added as a dependency to a non-existent unit containerd.service.
configure nydus snapshotter for containerd
Create /etc/containerd/config.toml.d
Drop-in the nydus configuration
[proxy_plugins]
  [proxy_plugins.nydus]
	type = "snapshot"
	address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
Restarting containerd
Failed to restart containerd.service: Unit containerd.service is masked.

Expected Behavior

When INSTALL_NYDUS_SNAPSHOTTER: "false" is set in the CcRuntime resource, the pre-install daemon should:

• Not attempt to install or configure Nydus Snapshotter
• Skip any steps related to Nydus configuration
• Not attempt to restart containerd

Questions

• Is the Nydus Snapshotter installation currently required by the operator even when explicitly disabled?
• Is there a specific reason why the environment variable setting is being overridden or ignored?
• Could you provide guidance on how to properly use the operator with RKE2/K3s where containerd is embedded and managed differently?

Workarounds Attempted

• Setting INSTALL_NYDUS_SNAPSHOTTER: "false" in the CcRuntime resource
• Creating a dummy containerd.service file
• Bypassing the operator completely and manually installing Kata Containers

Some additional info

• Containerd configuration is stored in /var/lib/rancher/rke2/agent/etc/containerd/ instead of /etc/containerd/
• The containerd service is not directly managed by systemd as a standalone service
• The standard containerd.service is often masked to prevent conflicts

It would be beneficial if the operator could detect RKE2/K3s environments and adapt accordingly, or at minimum, respect the INSTALL_NYDUS_SNAPSHOTTER: "false" setting when provided.

[hoo29]

I am hacking around but if it helps anyone else, I got the AMD SEV-SNP example working with RKE2 by doing:

1. Install fedora 42 cloud edition
2. dnf install fuse
3. Install rke2
4. ln -s /var/lib/rancher/rke2/agent/etc/containerd /etc/containerd
5. Add Alias=containerd.service under the [Install] section in /usr/lib/systemd/system/rke2-server.service. Then daemon reload, re-enable the service and restart it.
6. Install CoCo following https://confidentialcontainers.org/docs/getting-started/installation/. The deploy operator stage will fail to finish.
7. Edit /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl and change the imports line to be imports = ["/opt/kata/containerd/config.d/kata-deploy.toml", "/var/lib/rancher/rke2/agent/etc/containerd/config.toml.d/nydus-snapshotter.toml"]. I ended with up with 5 identical import kata-deploy.toml somehow.
8. Restart rke2.
9. Deploy the example workload in https://confidentialcontainers.org/docs/examples/snp-container-launch/ (which has a typo in the yaml).