AWS Secret Manager support

[jeffmccune]

Support writing secrets to AWS secret manager. Use case is for a management cluster in a management account to write secrets to AWS SM in a non-prod account developers have access to. And a prod account developers do not have access to. Leverage high quality open source CNCF projects like the external-secrets-operator. Minimize code in holos-console to minimize support and upkeep of this service.

The initial idea from the Platform Engineering team is to use a PushSecret and have the holos-console store a reference to the PushSecret, then update an annotation on the PushSecret when a Secret is updated, triggering the sync to AWS SM. Evaluate the pros and cons of this approach.

Assume the user has Pod Identity configured and working well.

---

Design

Overview

Integrate holos-console with the External Secrets Operator (ESO) PushSecret CRD to sync K8s Secrets to AWS Secrets Manager. holos-console acts as a thin coordination layer: it stores references to PushSecret resources, bumps a force-sync annotation when secrets change, and reads PushSecret status conditions to surface sync state in the UI.

Architecture

flowchart LR
    HC[holos-console<br/>Go server]
    KS[K8s Secret]
    PS[PushSecret<br/>ESO CRD]
    ESO[ESO Controller]
    AWS[AWS Secrets<br/>Manager]

    HC -- "K8s Secret CRUD" --> KS
    HC -- "Annotation bump<br/>force-sync timestamp" --> PS
    HC -. "Read status<br/>conditions" .-> PS
    KS -- "spec.selector<br/>references" --- PS
    ESO -- "Watches" --> PS
    ESO -- "Reads data" --> KS
    ESO -- "PutSecretValue" --> AWS

Loading

How It Works

1. Platform engineer pre-creates SecretStore (or ClusterSecretStore) and PushSecret resources via GitOps. The PushSecret's spec.selector.secret.name references a K8s Secret managed by holos-console.
2. Platform engineer annotates the holos-console K8s Secret with console.holos.run/push-secrets: '["my-push-secret"]' to register the association. This can also be done via the UI (Phase 4).
3. User creates or updates a secret via holos-console.
4. holos-console reads the push-secrets annotation, finds the referenced PushSecret resources, and patches each with console.holos.run/force-sync: "<unix-timestamp>".
5. ESO PushSecret controller detects the resource version change on the PushSecret (it watches PushSecret resources), triggers immediate reconciliation, reads the source K8s Secret, and pushes it to AWS SM.
6. User sees push status in the UI (synced, pending, error) by reading PushSecret status conditions.

Storage Model

On the K8s Secret (managed by holos-console):

console.holos.run/push-secrets: '["my-push-secret","another-push-secret"]'

JSON array of PushSecret resource names in the same namespace.

On the PushSecret (managed by ESO, annotation added by holos-console):

console.holos.run/force-sync: "1706900000"

Unix timestamp set by holos-console to trigger reconciliation.

Approach Evaluation

Proposed approach: PushSecret annotation bump

Pros:

• Minimal code in holos-console — ~200-300 lines of Go (annotation read, dynamic client patch, status read). No AWS SDK dependency.
• Leverages ESO, a mature CNCF project, for all AWS API interactions, retries, error handling, and credential management.
• Pod Identity works seamlessly — ESO uses the service account's IAM role via Pod Identity. No credentials stored in holos-console.
• Multi-account support — Multiple PushSecrets can reference the same source secret, each targeting a different SecretStore (non-prod account, prod account). Platform engineers control topology via GitOps.
• Immediate sync — The PushSecret controller watches for resource version changes. Patching an annotation changes the resource version, triggering near-immediate reconciliation (~seconds).
• Consistent with existing patterns — holos-console already uses annotations for all metadata (grants, description, URL). Adding a push-secrets annotation is natural.
• Clean separation of concerns — holos-console manages secrets + RBAC, ESO handles cloud provider sync. Each component does what it's best at.

Cons:

• Requires ESO installed and configured — Acceptable prerequisite for the use case. ESO is already a standard component in CNCF-aligned platform stacks.
• Sync errors surface indirectly — PushSecret sync failures appear in ESO status conditions, not natively in holos-console. The GetPushStatus RPC bridges this gap but adds a secondary read path.
• Annotation references can drift — If a PushSecret is renamed or deleted outside holos-console, the annotation becomes stale. Mitigated by: (a) status check will show errors or "not found" for stale references, (b) UI can surface stale references for cleanup.
• Small sync latency — Annotation bump → controller watch event → reconcile → AWS API call. Typically completes within seconds, but not instantaneous.

Alternatives considered and rejected:

1. Direct AWS SDK integration — Violates the "minimize code" goal. Would require maintaining AWS SDK dependency, multi-account assume-role logic, retry handling, and credential management — all of which ESO already handles.
2. Custom Kubernetes controller — More code to maintain than the annotation-bump approach. ESO's PushSecret controller already implements the reconciliation loop with proper error handling and status reporting.
3. ExternalSecret (pull direction) — Wrong direction. ExternalSecret pulls from external stores into K8s. We need push from K8s to AWS SM.
4. holos-console creates PushSecrets — Over-engineers the solution. Platform engineers should define push topology (which secrets go where, which accounts) via their standard GitOps workflow. holos-console just triggers syncs.

RBAC

Push operations inherit the same RBAC as the underlying secret operation:

• Trigger push: Implicit on CreateSecret and UpdateSecret — if you can write the secret, the push triggers automatically.
• View push status: Requires PERMISSION_SECRETS_READ (same as reading secret data). Returned via GetPushStatus RPC.
• Manage push associations (add/remove PushSecret references): Requires PERMISSION_SECRETS_ADMIN (OWNER role). This is a metadata change analogous to updating sharing grants.

Prerequisites (not in scope of this issue)

• ESO installed in the cluster (external-secrets.io CRDs available).
• SecretStore or ClusterSecretStore configured for each target AWS account.
• PushSecret resources created referencing holos-console-managed K8s Secrets.
• Pod Identity configured so ESO can authenticate to AWS.

---

Implementation Plan

Phase 1: Protobuf Definitions

Extend the SecretsService protobuf with push status support.

Changes to proto/holos/console/v1/secrets.proto:

1. Add GetPushStatus RPC to SecretsService.
2. Add PushSecretStatus message with fields: name (string), ready (bool), message (string — error detail when not ready), last_synced_unix (int64).
3. Add GetPushStatusRequest message with fields: project (string), name (string — the K8s Secret name).
4. Add GetPushStatusResponse message with field: push_secrets (repeated PushSecretStatus).
5. Add push_secrets field (repeated string) to SecretMetadata to surface associated PushSecret names in list responses.
6. Add UpdatePushSecrets RPC, UpdatePushSecretsRequest (project, name, push_secrets repeated string), and UpdatePushSecretsResponse for managing the association annotation.
7. Run make generate.

Phase 2: Push Trigger Mechanism (RED → GREEN)

RED — Write failing tests:

• console/secrets/push_test.go: Test TriggerPushSecrets function:
  • When secret has no push-secrets annotation → no-op, no error.
  • When secret has push-secrets: '["ps1","ps2"]' → patches each PushSecret with force-sync annotation.
  • When a referenced PushSecret doesn't exist → logs warning, continues with remaining PushSecrets, returns nil (best-effort).
  • Annotation parsing error → logs warning, returns nil (best-effort).

GREEN — Implement:

1. console/secrets/push.go: New file with:

   • PushSecretsAnnotation constant: "console.holos.run/push-secrets"
   • ForceSyncAnnotation constant: "console.holos.run/force-sync"
   • GetPushSecretNames(secret *corev1.Secret) ([]string, error) — parses the annotation.
   • PushClient struct wrapping dynamic.Interface and *resolver.Resolver.
   • NewPushClient(dynamicClient dynamic.Interface, resolver *resolver.Resolver) *PushClient.
   • TriggerPushSecrets(ctx context.Context, project string, pushSecretNames []string) error — patches each named PushSecret with force-sync: <unix-timestamp> using the dynamic client. Best-effort: logs warnings for missing PushSecrets, does not fail the parent operation.
   • GetPushStatus(ctx context.Context, project string, pushSecretNames []string) ([]*consolev1.PushSecretStatus, error) — reads each PushSecret's .status.conditions via the dynamic client. Returns status for each.

2. Wire into Handler: Add pushClient *PushClient field (may be nil if dynamic client unavailable). After successful CreateSecret and UpdateSecret, call TriggerPushSecrets. Best-effort: log and continue if push trigger fails — never fail the secret write because of a push trigger error.

3. Wire into console.go: Create dynamic.NewForConfig(restConfig) alongside the existing kubernetes.NewForConfig(restConfig). Pass to PushClient. Pass PushClient to Handler.

Phase 3: Push Status RPC (RED → GREEN)

RED — Write failing tests:

• console/secrets/handler_test.go: Test GetPushStatus RPC:

  • Returns UNAUTHENTICATED without claims.
  • Returns INVALID_ARGUMENT without project or name.
  • Returns PERMISSION_DENIED without read access.
  • Returns empty list when secret has no push-secrets annotation.
  • Returns status for each referenced PushSecret.
  • Returns ready: false with message when PushSecret not found.

• console/secrets/handler_test.go: Test UpdatePushSecrets RPC:

  • Returns UNAUTHENTICATED without claims.
  • Returns PERMISSION_DENIED without admin access.
  • Updates the push-secrets annotation on the K8s Secret.
  • Validates PushSecret names are non-empty strings.

GREEN — Implement:

1. GetPushStatus handler method: Extract push-secrets annotation from the K8s Secret, call pushClient.GetPushStatus, return results. RBAC: require PERMISSION_SECRETS_READ.

2. UpdatePushSecrets handler method: Update the push-secrets annotation on the K8s Secret. RBAC: require PERMISSION_SECRETS_ADMIN (OWNER).

3. Populate push_secrets in SecretMetadata: In buildSecretMetadata, read the annotation and populate the push_secrets string list so the UI gets this data from ListSecrets.

Phase 4: UI Integration

1. SecretPage — Push status indicator: Below the secret name/description area, show a "Push Status" section when metadata.push_secrets is non-empty. For each PushSecret, display:

   • Name
   • Status chip: green "Synced" when ready, yellow "Pending" when not ready and no error, red "Error" with message when not ready with error.
   • Last synced timestamp (relative, e.g., "2 minutes ago").
   • Fetch status via GetPushStatus RPC on page load and after save.

2. SecretPage — Push secrets management: For OWNER users, add an "External Sync" section (or integrate into the existing sharing panel) where they can:

   • View associated PushSecret names.
   • Add/remove PushSecret references via UpdatePushSecrets RPC.
   • Input validation: PushSecret name must be a valid K8s resource name.

3. SecretsListPage — Push indicator: In the secrets list table, add a small sync icon/indicator column showing whether the secret has push targets and their aggregate status (all synced = green, any error = red, otherwise = yellow).

Phase 5: Final Cleanup

1. Scan for dead code, unused imports, stale comments introduced by the implementation.
2. Update AGENTS.md architecture section to document the push integration.
3. Verify make generate && make test && make lint all pass.
4. Remove any TODO comments that were resolved during implementation.

---

TODO

• Phase 1: Protobuf definitions
• Phase 2: Push trigger mechanism
• Phase 3: Push status RPC
• Phase 4: UI integration
• Phase 5: Final cleanup

[jeffmccune]

@manrueda WDYT of this AWS SM support design / plan?