Skip to content

feat(ha): Add leader election to prevent duplicate ACM imports#48

Open
starlightromero wants to merge 5 commits intorobertlestak:mainfrom
starlightromero:feat/leader-election
Open

feat(ha): Add leader election to prevent duplicate ACM imports#48
starlightromero wants to merge 5 commits intorobertlestak:mainfrom
starlightromero:feat/leader-election

Conversation

@starlightromero
Copy link
Copy Markdown
Contributor

@starlightromero starlightromero commented Apr 11, 2026

Summary

Add leader election so only one replica performs certificate syncs at a time, preventing duplicate ACM imports that burn through the 5,000 import quota.

Problem

With multiple replicas, each independently watches secrets and imports certificates to ACM. This causes:

  • 3x the ACM imports per renewal cycle
  • Burst throttling from concurrent imports
  • Duplicate ImportCertificate API calls counting against the rolling 365-day quota

Changes

  • Implement leader election using client-go LeaseLock in main.go
  • Extract informer logic into runController as the OnStartedLeading callback
  • Add Role/RoleBinding for coordination.k8s.io Lease access
  • Expose KubeConfig from state package for leader election client
  • Enabled by default; set LEADER_ELECTION_ENABLED=false to disable

Configuration

Env Var Default Description
LEADER_ELECTION_ENABLED true Enable/disable leader election
LEADER_ELECTION_NAMESPACE Pod namespace Namespace for the Lease resource
LEADER_ELECTION_LOCK_NAME cert-manager-sync-leader Name of the Lease resource

Helm values

leaderElection:
  enabled: true
  lockName: cert-manager-sync-leader

Implement leader election using client-go LeaseLock so only one
replica performs certificate syncs at a time. Other replicas remain
as hot standbys for failover.

- Extract informer logic into runController as OnStartedLeading
  callback
- Add Role/RoleBinding for coordination.k8s.io Lease access
- Expose KubeConfig from state package for leader election client
- Configurable via LEADER_ELECTION_ENABLED, LEADER_ELECTION_NAMESPACE,
  and LEADER_ELECTION_LOCK_NAME env vars
- Enabled by default; set LEADER_ELECTION_ENABLED=false to disable
Update vulnerable dependencies:
- go-jose/v4: 4.1.3 -> 4.1.4 (CVE-2026-34986, HIGH)
- grpc: 1.79.1 -> 1.79.3 (CVE-2026-33186, CRITICAL)

Add tests for leader election configuration and KubeConfig
export from state package.
- Write TLS private keys with 0600 instead of 0644 (G306)
- Add read/write/idle timeouts to metrics HTTP server (G114)
- Annotate intentional 0644 on public cert/CA files
- G115: Integer overflow guarded by retries < 31 bounds check
- G703: Standard k8s KUBECONFIG path resolution pattern
- G117: Secret marshaling required by Imperva/ThreatX APIs
- G104: Handle AddEventHandler return value

Reduces gosec findings from 5 to 0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant