Feature Request: Support native ClusterProfile integration (KEP-5339) in Project Sveltos

[kahirokunn]

Summary

First of all, thank you for building and maintaining such a wonderful product!
Sveltos is a truly great project, and we really appreciate the value it brings to the multi-cluster community. 🙏✨

We would like to humbly propose adding native integration with the ClusterProfile API (SIG Multicluster) and its credential provider plugin model (KEP-5339).

Today Sveltos does not consume ClusterProfile, so it can’t leverage the standardized status.credentialProviders and the emerging Cluster Inventory ecosystem. By adopting ClusterProfile + KEP-5339, Sveltos could connect to managed clusters using short-lived credentials obtained via a pluggable exec flow (aligned with kubeconfig’s external credential provider protocol).

This one change could potentially address—and allow closing—the following issues all at once:

• Feature Request: Support for Multi-Cluster ClusterProfiles #430
• Feature Request: Support exec plugin in Sveltos #445
• Feature Request: Support ControllerIdentity in Sveltos similar to CAPA #440

Background (why this is a re-proposal)

Previously, I opened an earlier proposal for ClusterProfile integration. At that time, ClusterProfile did not yet have the credentialProviders specification; it could only hold kubeconfigs. In that state, there was no meaningful functional difference between a SveltosCluster resource and a ClusterProfile, so the integration was understandably declined.

However, the situation has since changed significantly:

• credentialProviders has been added to ClusterProfile, and implementations are landing.
• At recent events such as KubeCon, we observed engineers from providers like GCP and AWS actively aligning with ClusterProfile.
• This signals that ClusterProfile is becoming the upstream standard point of integration for credentials and inventory.

Given this, we believe that following this evolution early would carry meaningful benefits for Sveltos in terms of ecosystem alignment, security posture, and broader adoption. It could help Sveltos capture an even stronger share in the multi-cluster management space.

With the greatest respect, we would like to kindly re-propose native ClusterProfile integration, this time backed by the new credentialProviders functionality and its supporting ecosystem.

Why this matters (Motivation)

• Ecosystem alignment: ClusterProfile is becoming the common substrate for multi-cluster inventory and access. Without compatibility, Sveltos risks divergence from upstream conventions and tooling.
• Security posture: KEP-5339 places no secrets in ClusterProfile. Credentials are obtained out-of-tree via an exec plugin chosen at deploy time, enabling short-lived tokens, SSO/WIF/OIDC, and provider-agnostic flows.
• Operational flexibility: The plugin model avoids baking cloud-specific SDKs into Sveltos. Platform teams can pick/upgrade credential plugins independently of Sveltos releases.
• Ease of integration: There is a Go library implementation in the Cluster Inventory project designed to construct the cluster connection info from ClusterProfile.status and invoke credential plugins. This keeps Sveltos changes minimal while providing maximum interoperability.
  → Implementation PR: Define plugin interface and consumer library

Suggested direction (High-level)

Would you consider:

• Adopting the ClusterProfile API as a first-class inventory source in Sveltos?
• Honoring status.credentialProviders to discover the cluster’s accepted credential type(s) and connection parameters?
• Using the provided Go package from the Cluster Inventory project to obtain short-lived credentials via the exec plugin protocol (KEP-541 compatible I/O), then building the client config to talk to the target cluster?

Note: This request is intentionally framed as an integration and alignment proposal, not prescribing a specific internal design or plugin set. The goal is to let Sveltos naturally plug into the Cluster Inventory & ClusterProfile ecosystem and benefit from provider-maintained exec plugins.

Potential benefits to Project Sveltos

• Closes three open asks at once: could unblock Feature Request: Support for Multi-Cluster ClusterProfiles #430, Feature Request: Support exec plugin in Sveltos #445, Feature Request: Support ControllerIdentity in Sveltos similar to CAPA #440 by standardizing how Sveltos discovers clusters and obtains credentials.
• Future-proofing: aligns with upstream multi-cluster APIs and avoids bespoke inventory/cred flows.
• Provider-neutral: works equally well with GKE, EKS, AKS, on-prem, or any environment that supplies an exec credential plugin.
• Cleaner UX: users point Sveltos at ClusterProfiles; Sveltos handles the rest through the standard plugin interface.

Backward compatibility

• This could be added alongside current Sveltos mechanisms. Users can opt into ClusterProfile-based integration without breaking existing flows.

Open questions (for discussion)

• Which credential types should be “known” by default vs. entirely user-configured?
• How should Sveltos surface errors/observability around plugin execution and token acquisition?
• What’s the minimal surface to declare ClusterProfile support (read-only of status vs. deeper interactions)?

References

• KEP-5339: Plugin for Credentials in ClusterProfile
• Define plugin interface and consumer library by qiujian16 · Pull Request #17 · kubernetes-sigs/cluster-inventory-api
• Slack thread
• KEP-4322: ClusterProfile API
• ClusterProfile API Overview - SIG Multicluster
• [PUBLIC] ClusterProfile Credentials via Plugin - Google Slides

[kahirokunn]

One practical note as you consider this proposal: while KEP-5339 documents the exec-based credential plugin model and provides the SDK-level primitives, it does not ship concrete plugin implementations or binaries. That can make hands-on validation a bit tricky in the short term.
To make evaluation easier, I’ve put together a small public repository with sample ClusterProfile Credential Plugins and straightforward instructions to try the exec flow end-to-end:
labthrust/clusterprofile-credentials-plugins
https://github.com/labthrust/clusterprofile-credentials-plugins/tree/main
This is intended purely as a testing harness and reference so it’s easier to assess whether the approach fits Sveltos. I hope it’s helpful during your exploration. I’m very happy to adapt or extend it based on your feedback (e.g., plugin shapes, error/observability hooks, or anything else that would make evaluation smoother).
Thank you again for all the thoughtful work on Sveltos and for considering this integration.

[gianlucam76]

@kahirokunn what is not clear to me is how to get the credentials to access a cluster.

Integration with ClusterAPI is easy because:

1. A ClusterAPI cluster exists in the management cluster (so labels can be added)
2. A Kubeconfig to access the managed cluster is available in the management cluster (so Sveltos can take it and build a client to access the remote cluster)

Same applies to Claudie for instance and for it I did build a controller https://github.com/gianlucam76/claudie-sveltos-integration

If those two conditions applies here (provided this integration can be tested with a kind cluster) I am happy to move this forward. But I need both conditions to be satisfied.

[kahirokunn]

@gianlucam76 Thank you for your questions. 🙏
Let me address your two conditions in detail:

Regarding Condition 1: "A ClusterAPI cluster exists in the management cluster (so labels can be added)"

This is fully YES. ClusterProfile custom resources exist in the management cluster. The ClusterProfile is created by a cluster manager (not Sveltos itself, but another controller or human operator).

For testing purposes, you could create one like this:

kubectl --context "kind-hub" apply -f - <<EOF
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
  name: spoke-1
  namespace: default
spec:
  clusterManager:
    name: demo
EOF

STATUS_PATCH=$(cat <<EOF
{
  "status": {
    "credentialProviders": [
      {
        "name": "secretreader",
        "cluster": {
          "server": "${SERVER}",
          "certificate-authority-data": "${CADATA}",
          "extensions": [
            {
              "name": "client.authentication.k8s.io/exec",
              "extension": {
                "clusterName": "spoke-1"
              }
            }
          ]
        }
      }
    ]
  }
}
EOF
)

kubectl --context "kind-hub" patch clusterprofile "spoke-1" --type=merge --subresource=status -p "${STATUS_PATCH}"

Of course, you can freely add labels to it as needed.

Regarding Condition 2: "A Kubeconfig to access the managed cluster is available in the management cluster"

This can be both YES and NO - it depends on the situation. I'm hoping we can support both cases for compatibility.

The situation is:

• There might be secrets in the cluster containing credentials
• There might not be any secrets in the cluster at all

This is determined by the credential provider plugin. Multiple credential provider plugins can be available. The mechanism is the same as exec plugins used in kubeconfig. I'm currently working on a PR for the official sample credential provider plugins:

kubernetes-sigs/cluster-inventory-api#21

How the mechanism works to access Kubernetes clusters

1. Binary placement on controller containers

The plugin binaries need to be placed on the filesystem of controller containers that access managed clusters via ClusterProfile (in Sveltos' case, this would include Event Manager, AddOn Controller, Agent, etc.).

This is the user's responsibility, not Sveltos', though guidelines would be helpful.

One pattern could be using an initContainer to download single-binary plugins like secretreader-plugin to a shared volume for subsequent containers to use.

2. Provider configuration file

A configuration file like this needs to be available to all controllers accessing managed clusters via ClusterProfile:

{
  "providers": [
    {
      "name": "secretreader",
      "execConfig": {
          "apiVersion": "client.authentication.k8s.io/v1",
          "command": "/bin/secretreader-plugin",
          "provideClusterInfo": true
      }
    }
  ]
}

This is the user's responsibility, not Sveltos', though guidelines would be helpful.

3. Controller startup with flag

Controllers need to be started with the -clusterprofile-provider-file argument pointing to the mounted ConfigMap file:

KUBECONFIG=./examples/controller-example/hub.kubeconfig ./examples/controller-example/controller-example.bin \
  -clusterprofile-provider-file ./examples/controller-example/cp-creds.json

4. Plugin-specific permissions

Each plugin has different permission requirements. For example:

For secretreader plugin - needs permission to read secrets containing managed cluster access tokens:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secretreader
  namespace: <CONTROLLER_NAMESPACE>
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secretreader
  namespace: <CONTROLLER_NAMESPACE>
subjects:
- kind: ServiceAccount
  name: <CONTROLLER_SERVICE_ACCOUNT_NAME>
  namespace: <CONTROLLER_NAMESPACE>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: secretreader

For eks-aws-auth-plugin (as an example I created at https://github.com/labthrust/clusterprofile-credentials-plugins/tree/main/cmd/eks-aws-auth-plugin), controllers accessing EKS clusters need IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EksDiscover",
      "Effect": "Allow",
      "Action": [
        "eks:ListClusters",
        "eks:DescribeCluster"
      ],
      "Resource": "*"
    }
  ]
}

Here's a Terraform example for reference:

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }

    actions = [
      "sts:AssumeRole",
      "sts:TagSession",
    ]
  }
}

data "aws_iam_policy_document" "eks_discover" {
  statement {
    sid       = "EksDiscover"
    effect    = "Allow"
    actions   = ["eks:ListClusters", "eks:DescribeCluster"]
    resources = ["*"]
  }
}

resource "aws_iam_role" "example" {
  name               = "eks-pod-identity-example"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

resource "aws_iam_role_policy" "eks_discover" {
  name   = "eks-discover"
  role   = aws_iam_role.example.name
  policy = data.aws_iam_policy_document.eks_discover.json
}

resource "aws_eks_pod_identity_association" "projectsveltos" {
  # access managed cluster by clusterprofile api controllers
  for_each = toset([
    "addon-controller",
    "event-manager",
    "classifier-manager",
    "sveltos-agent-manager",
  ])
  cluster_name    = aws_eks_cluster.example.name
  namespace       = "projectsveltos"
  service_account = each.value
  role_arn        = aws_iam_role.example.arn
}

This is the user's responsibility, not Sveltos', though guidelines would be helpful.

What Sveltos would need to focus on

The key points for Sveltos are remarkably simple:

1. Parse the flag
2. Get the ClusterProfile
3. Pass the ClusterProfile to cpCreds.BuildConfigFromCP(cp) to get rest.Config - the plugin resolution mechanism is entirely handled by cpCreds.BuildConfigFromCP

Here's the controller implementation example from [the PR](https://github.com/kubernetes-sigs/cluster-inventory-api/pull/21/files#diff-5105a48d80c60db46a04513f7e263c0edcbb859dc2a9ab8e63177e85532010c8):

package main

import (
	"context"
	"flag"
	"log"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	k8sclient "k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/clientcmd"
	ciaclient "sigs.k8s.io/cluster-inventory-api/client/clientset/versioned"
	"sigs.k8s.io/cluster-inventory-api/pkg/credentials"
)

func main() {
	// Flags
	credentialsProviders := credentials.SetupProviderFileFlag()
	namespace := flag.String("namespace", "default", "Namespace of the ClusterProfile on the hub cluster")
	clusterProfileName := flag.String("clusterprofile", "", "Name of the ClusterProfile to target (required)")
	flag.Parse()

	if *clusterProfileName == "" {
		log.Fatalf("-clusterprofile is required")
	}

	// Load providers file
	cpCreds, err := credentials.NewFromFile(*credentialsProviders)
	if err != nil {
		log.Fatalf("Got error reading credentials providers: %v", err)
	}

	// Build hub client and get ClusterProfile
	loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
	configOverrides := &clientcmd.ConfigOverrides{}
	hubClientConfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, configOverrides)
	hubConfig, err := hubClientConfig.ClientConfig()
	if err != nil {
		log.Fatalf("failed to load default kubeconfig for hub: %v", err)
	}
	cic, err := ciaclient.NewForConfig(hubConfig)
	if err != nil {
		log.Fatalf("failed to construct cluster-inventory client: %v", err)
	}
	cp, err := cic.ApisV1alpha1().ClusterProfiles(*namespace).Get(context.Background(), *clusterProfileName, metav1.GetOptions{})
	if err != nil {
		log.Fatalf("failed to get ClusterProfile %s/%s: %v", *namespace, *clusterProfileName, err)
	}

	// Build rest.Config for the spoke cluster using the credential provider
	spokeConfig, err := cpCreds.BuildConfigFromCP(cp)
	if err != nil {
		log.Fatalf("Got error generating spoke rest.Config: %v", err)
	}

	// Create a Kubernetes client for the spoke cluster and list pods
	mclient, err := k8sclient.NewForConfig(spokeConfig)
	if err != nil {
		log.Fatalf("failed to create spoke client: %v", err)
	}
	plist, err := mclient.CoreV1().Pods("").List(context.Background(), metav1.ListOptions{})
	if err != nil {
		log.Fatalf("failed to list pods on spoke: %v", err)
	}
	log.Printf("Listed %d pods on spoke cluster", len(plist.Items))
	for _, p := range plist.Items {
		log.Printf("pod: %s/%s", p.Namespace, p.Name)
	}
}

Summary of required changes

Based on this, here's what would be needed:

For each controller:

1. Parse the flag
2. Get the ClusterProfile
3. Use cpCreds.BuildConfigFromCP(cp) to obtain rest.Config and operate with it

For Helm charts:

1. Provide a method for users to freely download plugins via initContainer + ephemeral volume
2. Enable passing the -clusterprofile-provider-file flag to controllers
3. Add RBAC for CRUD operations on ClusterProfile custom resources

Alternative approach

I'm also considering another pattern: creating a custom controller that watches ClusterProfile resources and creates corresponding SveltosCluster + kubeconfig resources, then rotates the tokens. This approach would mean:

• Not all controllers need to support ClusterProfile directly
• Configuration cannot be as granular
• But it would be simpler to implement

Both approaches have their merits, and I wanted to present both options for your consideration.

Thank you again for your time and consideration in exploring this integration!

[gianlucam76]

Thank you. The custom controller feels better to start. I did something similar to support Claudie

https://github.com/gianlucam76/claudie-sveltos-integration

As you mentioned a controller that watches for ClusterProfile and creates SveltosCluster + kubeconfig resources and keeps token valid by refreshing it.

So only place where plugins are needed is this. potentially if this combination gains popularity, we can move this component directly into Projectsveltos (though I would start with repo outside of projectsveltos org so whoever wants to contribute can)

What do you think?

[kahirokunn]

I completely agree with your suggestion. Starting with a dedicated controller that watches ClusterProfile and generates corresponding SveltosCluster and secrets for rotation feels like the most practical path forward.

That said, I think there are multiple ways such extensions can begin. To ensure quality and consistency—and to make reviews easier for contributors—it might be very helpful if the Projectsveltos organization provided an official GitHub template repository as a model case.

Such a template (in Go, bundled with Helm) would give users a ready-made structure where the only part they need to implement is a function that returns cluster connection info and tokens. From there, the controller scaffolding would handle creating/updating SveltosCluster objects and the associated secrets, including rotation.

On further thought, though, I wonder if a lightweight SDK or framework (or both) might be even better than a one-off template. That would allow contributors to easily follow evolving best practices while keeping their implementations aligned with updates from the upstream model cases.

That’s the direction I’m currently imagining.
what do you think? 👀

[gianlucam76]

I can take the task to create an empty repository that follows Sveltos repos (all based on kubebuilder and few tweaks). So that can be eventually moved to Projectsveltos org if there is enough traction.

But I dont have enough knowledge on ClusterProfile API (SIG Multicluster) to implement it (though I can help with anything related to Sveltos)

[kahirokunn]

Thank you so much. that would already be a huge help. 🙏
No need to include anything about the ClusterProfile API itself.

If you could kindly create just the empty repository that follows the usual Sveltos repo structure (kubebuilder + your standard tweaks), that would be more than enough for us to build upon.

I really appreciate your support!

[ocraviotto]

Thank you both, this thread was helpful in helping me to figure out a way to register a cluster created with KRO + Config Connector in GCP while using a DNS-based endpoint to access to the GKE control plane. I thought I'd share how I did that in case someone else might find it useful in the meantime.

My work around was to use External Secrets (which I already had in the management cluster) via its GCRAccessToken custom resource to get the token, and then ExternalSecret to render the secret with the kubeconfig and keep it fresh via refreshInterval.
This did require to give ESO's service account roles/container.admin permissions via workload identity (I could further tailor permissions via a custom role), but once done, it worked as intended and I was able to have Sveltos do its thing.

I would expect that the same approach could be adapted to other Generators.

Sharing the full ResrouceGraphDefinition below:

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: sveltosclusterregistration.kro.run
spec:
  schema:
    apiVersion: v1alpha1
    kind: SveltosGKEClusterRegistration
    spec:
      name: string | required=true
      projectId: string | required=true
      parentProjectId: string | required=true
      parentClusterLocation: string | required=true
      parentClusterName: string | required=true
      tokenKSAName: string | required=true
      clusterEndpoint: string | required=true
      clusterLocation: string | required=true
      environment: string | default=staging

  resources:
  - id: tokenGSAServiceAccount
    template:
      apiVersion: iam.cnrm.cloud.google.com/v1beta1
      kind: IAMServiceAccount
      metadata:
        annotations:
          cnrm.cloud.google.com/project-id: ${schema.spec.projectId}
        name: ${schema.spec.tokenKSAName}

  - id: tokenGSAIamPolicyMember
    template:
      apiVersion: iam.cnrm.cloud.google.com/v1beta1
      kind: IAMPolicyMember
      metadata:
        name: token-sa-${schema.spec.name}-policybinding
      spec:
        member: ${tokenGSAServiceAccount.status.member}
        role: roles/container.admin
        resourceRef:
          apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
          kind: Project
          external: projects/${schema.spec.projectId}

  - id: tokenKSA
    template:
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        annotations:
          iam.gke.io/gcp-service-account: ${tokenGSAServiceAccount.status.email}
        name: ${schema.spec.tokenKSAName}

  - id: tokenKSAtoGSAbinding
    template:
      apiVersion: iam.cnrm.cloud.google.com/v1beta1
      kind: IAMPolicyMember
      metadata:
        name: token-sa-${schema.spec.name}-workloadidentity
      spec:
        member: serviceAccount:${schema.spec.parentProjectId}.svc.id.goog[${schema.spec.name}/${schema.spec.tokenKSAName}]
        role: roles/iam.workloadIdentityUser
        resourceRef:
          kind: IAMServiceAccount
          external: ${tokenGSAServiceAccount.status.name}
  
  - id: clusterAccessTokenGenerator
    template:
      # Using a workaround with GCRAccessToken to get the token we can use with Sveltos...
      apiVersion: generators.external-secrets.io/v1alpha1
      kind: GCRAccessToken
      metadata:
        name: ${schema.spec.name}
      spec:
        projectID: ${schema.spec.projectId}
        auth:
          workloadIdentity:
            serviceAccountRef:
              name: ${schema.spec.tokenKSAName}
            # Management cluster lives in a different project than child
            clusterLocation: ${schema.spec.parentClusterLocation} # The management cluster GCP location
            clusterName: ${schema.spec.parentClusterName} # The management cluster name in GCP
            clusterProjectID: ${schema.spec.parentProjectId} # The management cluster project ID

  - id: externalSecret
    template:
      # Sveltos requires a Kubeconfig like this
      apiVersion: external-secrets.io/v1
      kind: ExternalSecret
      metadata:
        name: ${schema.spec.name}-sveltos-kubeconfig
      spec:
        refreshInterval: "30m"
        target:
          template:
            type: Opaque
            data:
              kubeconfig: |
                apiVersion: v1
                kind: Config
                clusters:
                - name: ${schema.spec.name}
                  cluster:
                    server: https://${schema.spec.clusterEndpoint}
                users:
                - name: projectsveltos
                  user:
                    token: {{ .password }}
                contexts:
                - name: sveltos-context
                  context:
                    cluster: ${schema.spec.name}
                    namespace: ${schema.spec.name}
                    user: projectsveltos
                current-context: sveltos-context
        dataFrom:
          - sourceRef:
              generatorRef:
                apiVersion: generators.external-secrets.io/v1alpha1
                kind: GCRAccessToken
                name: ${clusterAccessTokenGenerator.metadata.name}

  - id: sveltosCluster
    template:
      apiVersion: lib.projectsveltos.io/v1beta1
      kind: SveltosCluster
      metadata:
        name: ${schema.spec.name}
        labels:
          platform: gke
          cluster: ${schema.spec.name}
          region: ${schema.spec.clusterLocation}
          environment: ${schema.spec.environment}