Add rollback controller reconcile loop

[IsaacJames]

Rollback Controller Implementation

This PR implements the RollbackReconciler which manages Rollback resources by triggering deployments via a configurable CICD backend and polling for completion.

internal/controller/deploy/rollback_controller.go

• Reconciliation loop:
  • Watches Rollback resources and triggers deployments via the injected Deployer
  • Polls for deployment status until completion (succeeded/failed)
  • Tracks attempt count with configurable retry limit
  • Updates rollback status conditions (InProgress, Succeeded)
  • Finds the currently active release to populate FromReleaseName

cmd/rollback-manager/main.go

• Configurable CICD backend via --cicd-backend flag (noop, github)
• GitHub token via --github-token flag / ROLLBACK_MANAGER_GITHUB_TOKEN env var
• Rollback history limit via --rollback-history-limit flag

Reconcile Loop

┌─────────────────────────────────────────────────────────────────────────────┐
│                           Rollback Reconcile Loop                           │
└─────────────────────────────────────────────────────────────────────────────┘

                              ┌──────────────┐
                              │   Rollback   │
                              │   Created    │
                              └──────┬───────┘
                                     │
                                     ▼
                        ┌────────────────────────┐
                        │  Succeeded condition   │───Yes───▶ Done (skip)
                        │       exists?          │
                        └────────────┬───────────┘
                                     │ No
                                     ▼
                        ┌────────────────────────┐
                        │  Fetch target Release  │───NotFound───▶ markRollbackFailed
                        └────────────┬───────────┘
                                     │ Found
                                     ▼
                        ┌────────────────────────┐
                        │  InProgress == True?   │
                        └────────────┬───────────┘
                                     │
                    ┌────────────────┴────────────────┐
                    │ No                              │ Yes
                    ▼                                 ▼
        ┌───────────────────────┐       ┌───────────────────────┐
        │  triggerDeployment()  │       │ pollDeploymentStatus()│
        └───────────┬───────────┘       └───────────┬───────────┘
                    │                               │
                    ▼                               ▼
        ┌───────────────────────┐       ┌───────────────────────┐
        │  AttemptCount >= 3?   │       │   Deployment Status   │
        └───────────┬───────────┘       └───────────┬───────────┘
                    │                               │
        ┌───────────┴───────────┐       ┌───────────┼───────────┬───────────┐
        │ Yes                   │ No    │ Succeeded │ Failed    │ Pending/  │
        ▼                       ▼       ▼           ▼           │ InProgress│
   markRollback           Call CICD  markRollback  Retry?       ▼           
   Failed()               Deployer   Succeeded()     │      Requeue         
                              │                      │      (15s)           
                              ▼               ┌──────┴──────┐               
                    Set InProgress=True       │ Yes         │ No            
                    Requeue (15s)             ▼             ▼               
                                        triggerDeployment  markRollback     
                                        (immediate retry)  Failed()         

Error Handling

• Retryable errors (5xx, rate limits): Requeue after 15s, increment attempt count
• Non-retryable errors: Mark rollback as failed immediately
• Max retries exceeded: Mark rollback as failed after 3 attempts

Rollback Status Conditions

Condition	Status	Meaning
InProgress	True	Deployment is running
InProgress	False + Succeeded	Terminal state reached
Succeeded	True	Rollback completed successfully
Succeeded	False	Rollback failed (all retries exhausted)

Manager Configuration

--cicd-backend       CICD backend to use (noop, github) [default: noop]
--github-token       GitHub API token (required for github backend)

[goelozev]

Looks good!

[goelozev]

👍

[goelozev]

Fabulous!

[0x0013]

Nicely done!

Added a couple comments.

[0x0013]

we're setting the condition here, but it seems it will get overwritten almost immediately in triggerDeployment.

[IsaacJames]

Not necessarily - if triggerDeployment fails to trigger the deployment with a retryable error, the condition won't be updated and the Rollback will get requeued for reconciliation. That said, I'm now wondering if the rollback.Status.Message message set on line 144 would be better set on the InProgress condition.

[IsaacJames]

@0x0013 I've decided to simplify this a bit and agree it doesn't make sense to set the InProgress=True condition here in pollDeploymentStatus. Instead, we now set it once at the start of triggerDeployment with Reason=DeploymentTriggering and once at the end after a successful trigger with Reason=DeploymentTriggered.

This should make the behaviour clearer when a Rollback gets stuck trying and failing to trigger a deployment via the cicd backend, as the reason DeploymentTriggering will remain, whilst the failures themselves will be recorded in the rollback.Status.Message.

[0x0013]

Same as noted previously, the transition time should get set by setStatusCondition.

[IsaacJames]

I'm leaning towards leaving this (and the LastTransitionTime below) as set explicitly since this is the only transition the InProgress condition will make to False (and likewise below with the Succeeded condition to True).

The value I see is that by setting rollback.Status.CompletionTime and the LastTransitionTime of both conditions in this function to the same now value, it'll be clearer that the conditions transitioned to the terminal states as part of the completion of the rollback (particularly programmatically, if we ever need to compare the timestamps).

[0x0013]

Added a few more comments.

One thing I'm not fully convinced about, is setting the RollbackConditionInProgress with the "DeploymentTriggering" reason. While I understand the idea behind the behavior, I think I would prefer for RollbackConditionInProgress==True to mean that Deployment has already been triggered. It would make it clearer what the condition state implies.

However, I'm not yet fully convinced myself, and I will think a little more whether the current approach could be problematic down the line.

[0x0013]

I'm not sure of the intention of the LastHTTPCallTime field, but we only update it here on pending or in progress status. Should it not be updated if status is failed or successful?

I'm aware that a new triggered deployment would update the field as well, but I think other cases wouldn't.

[0x0013]

Suggested change

	if deployerErr, ok := err.(*cicd.DeployerError); ok && deployerErr.Retryable {
	var deployerErr *cicd.DeployerError
	if errors.As(err, &deployerErr) && deployerErr.Retryable {

This should have the same outcome, but is better practice, because errors.As() will evaluate any error in the chain, in case cicd.DeployerError had been wrapped before it's returned.