Auto-retry failed Tasks in spawner dedup logic

[axon-agent]

🤖 Axon Agent @gjkim42

Summary

When a Task created by a TaskSpawner fails, the spawner's dedup logic previously treated it the same as a succeeded task — the task name existed in the existingTasks map, so the work item was skipped on subsequent poll cycles. This meant:

• Cron spawners: A failed task blocked retries until TTL cleanup. For axon-fake-strategist (TTL: 864000s = 10 days), a single failure could waste 10 days. For axon-workers (TTL: 3600s), it wasted an hour.
• GitHub issue spawners: A failed task for an issue blocked any retry for that issue until TTL deletion.

Changes

cmd/axon-spawner/main.go:

• Track failed tasks separately in the dedup phase
• Include failed tasks in the newItems list (eligible for retry)
• Before creating a retry task, delete the old failed Task object (required because Task spec is immutable and Kubernetes doesn't allow name reuse)
• Handle NotFound gracefully in case TTL already cleaned it up

cmd/axon-spawner/main_test.go:

• Add TestRunCycleWithSource_FailedTaskRetriedImmediately — verifies failed task is deleted and recreated
• Add TestRunCycleWithSource_SucceededTaskNotRetried — verifies succeeded tasks remain deduplicated
• Add TestRunCycleWithSource_FailedTaskRetryRespectsMaxConcurrency — verifies retries still respect concurrency limits
• Update TestRunCycleWithSource_CompletedTasksDontCountTowardsLimit to use two succeeded tasks (the original used one failed task which is now retried by design)

Design decisions

Why unconditional retry (no retry limit)? This is the simplest correct behavior — the spawner already has maxTotalTasks to cap total task creation, and maxConcurrency to limit parallelism. A retry-specific limit would add complexity for little gain. Users who want bounded retries can use maxTotalTasks. A future retryPolicy field (#298) could add more sophisticated control.

Why delete-and-recreate instead of updating? Task spec has an immutability validation rule (self == oldSelf), so the failed task cannot be updated. Deleting and recreating with the same name preserves the spawner's naming convention.

Relates to

• Partially addresses Workflow: Self-development cron spawners lack resilience and quality controls #287 (problem 1: cron dedup prevents retry after failure — the only remaining unaddressed item from that issue)

Test plan

• go test ./cmd/axon-spawner/ -v — all 27 tests pass
• make verify — passes
• make build — compiles cleanly
• CI pipeline (build, verify, test, test-integration)

---

Summary by cubic

Automatically retries failed Tasks by detecting failures during spawner dedup, deleting the failed Task, and recreating it with the same name. This prevents cron and issue spawners from getting stuck until TTL cleanup and still honors maxConcurrency and maxTotalTasks.

• Bug Fixes
  • Track failed tasks separately during dedup and include them for retry.
  • Delete failed Task before recreate (spec is immutable); ignore NotFound.
  • Keep succeeded tasks deduped; retries respect concurrency and total limits.
  • Tests added for immediate retry, succeeded not retried, and concurrency respected. Partially addresses Workflow: Self-development cron spawners lack resilience and quality controls #287.

^{Written for commit cf6c3bf. Summary will update on new commits.}

[cubic-dev-ai]

No issues found across 2 files