-
Notifications
You must be signed in to change notification settings - Fork 207
feat(engine): add verification, cost tracking, and smarter completion #331
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
highimpact-dev
wants to merge
7
commits into
subsy:main
Choose a base branch
from
highimpact-dev:feat/engine-improvements
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
9345e09
chore(spec): create engine improvements spec with 8 steps
highimpact-dev 2cc0bb8
feat(engine): complete step 1 - verification gates
highimpact-dev fdeeb40
feat(engine): complete steps 2-3 - auto-commit defaults & model escal…
highimpact-dev 2dcca2e
feat(engine): complete steps 4-5 - cross-iteration context & completi…
highimpact-dev 1fb98f7
feat(engine): complete steps 6-8 - AC validation, cost tracking, para…
highimpact-dev 069d253
Merge branch 'main' into feat/engine-improvements
subsy 0c3e684
fix: address PR review feedback — pricing, injection, and docs
highimpact-dev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| spec_path: spec/20260222-engine-improvements | ||
| max_iterations: 50 | ||
| current_iteration: 2 | ||
| started_at: 2026-02-22T00:00:00Z | ||
| # Ralph pattern: Circuit breaker tracking | ||
| no_progress_count: 2 | ||
| error_count: 0 | ||
| last_completed_step: 5 | ||
| circuit_breaker: open | ||
| # Learning: Trace tracking | ||
| current_trace_path: null | ||
| traces_emitted: 0 | ||
| --- | ||
|
|
||
| # Spec Loop Active | ||
|
|
||
| Implementing: spec/20260222-engine-improvements | ||
|
|
||
| ## Exit Conditions (Dual-Gate) | ||
| 1. All steps in PLAN.md marked ✅ | ||
| 2. Completion promise output: `<promise>ALL_STEPS_COMPLETE</promise>` | ||
|
|
||
| **Both conditions required for clean exit.** | ||
|
|
||
| ## Circuit Breaker Triggers | ||
| - 3 iterations with no step completion → OPEN | ||
| - 5 iterations with repeated errors → OPEN | ||
|
|
||
| When circuit breaker opens, analyze and fix before continuing. | ||
| last_completed_step: 5 | ||
| 0 | ||
| last_completed_step: 5 | ||
| 0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
74 changes: 74 additions & 0 deletions
74
spec/20260222-engine-improvements/01-verification-gates/COMPLETED.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| /** | ||
| * ABOUTME: Completion summary for Step 1: Verification Gates. | ||
| */ | ||
|
|
||
| # Step 1: Verification Gates — COMPLETED | ||
|
|
||
| ## Summary | ||
|
|
||
| Implemented configurable post-completion verification commands that run after an agent signals `<promise>COMPLETE</promise>` but before the task is marked done in the tracker. If verification fails, the task is NOT marked complete and the engine retries with the verification error output injected into the next prompt. | ||
|
|
||
| ## Files Created | ||
|
|
||
| ### `src/engine/verification.ts` (NEW) | ||
| Verification gate runner with: | ||
| - `runVerification(cwd, config)` — runs all commands via `sh -c`, stops on first failure, returns `VerificationResult` | ||
| - `formatVerificationErrors(result)` — formats failures into readable multi-line string for prompt injection | ||
|
|
||
| ### `tests/engine/verification.test.ts` (NEW) | ||
| 11 tests covering: | ||
| - All commands pass → `result.passed === true` | ||
| - First command fails → stops, `result.passed === false` | ||
| - Timeout → `result.passed === false` | ||
| - Empty commands → `result.passed === true` (vacuously true) | ||
| - Format errors → readable multi-line string with command, exit code, stdout, stderr | ||
| - Only failed commands appear in formatted output | ||
|
|
||
| ## Files Modified | ||
|
|
||
| ### `src/config/types.ts` | ||
| - Added `VerificationConfig` interface with `enabled`, `commands`, `timeoutMs`, `maxRetries` | ||
| - Added `DEFAULT_VERIFICATION_CONFIG` constant | ||
| - Added `verification?: VerificationConfig` to both `StoredConfig` and `RalphConfig` | ||
|
|
||
| ### `src/engine/types.ts` | ||
| - Added `'verification:started' | 'verification:passed' | 'verification:failed'` to `EngineEventType` union | ||
| - Added `VerificationStartedEvent`, `VerificationPassedEvent`, `VerificationFailedEvent` interfaces | ||
| - Added new events to `EngineEvent` union type | ||
|
|
||
| ### `src/engine/index.ts` | ||
| - Imported `runVerification`, `formatVerificationErrors`, `DEFAULT_VERIFICATION_CONFIG` | ||
| - Added `lastVerificationErrors: string` and `verificationRetryMap: Map<string, number>` private fields | ||
| - Modified `buildPrompt()` to accept and pass `verificationErrors` into template context | ||
| - Modified `runIteration()` to clear `lastVerificationErrors` when no pending verification retries | ||
| - Inserted verification gate between completion detection and task completion marking: | ||
| - Emits `verification:started` event | ||
| - Runs all configured commands | ||
| - On pass: emits `verification:passed`, clears state | ||
| - On fail: emits `verification:failed`, stores errors for next prompt, suppresses completion | ||
| - On exhausted retries (`verificationRetries >= maxRetries`): skips gate and marks done | ||
|
|
||
| ### `src/templates/types.ts` | ||
| - Added `verificationErrors: string` to `TemplateVariables` | ||
|
|
||
| ### `src/templates/engine.ts` | ||
| - Added `verificationErrors?: string` to `ExtendedTemplateContext` | ||
| - Wires `verificationErrors` through `buildTemplateVariables()` | ||
|
|
||
| ### `src/templates/builtin.ts` | ||
| - Added `{{#if verificationErrors}}` block to `JSON_TEMPLATE` after `recentProgress` | ||
|
|
||
| ### `src/plugins/trackers/builtin/json/template.hbs` | ||
| - Added same `{{#if verificationErrors}}` block (reference copy) | ||
|
|
||
| ## Verification Results | ||
|
|
||
| ``` | ||
| bun run typecheck ✓ (no errors) | ||
| bun run build ✓ (bundled successfully) | ||
| bun test ✓ 3278 pass, 0 fail | ||
| ``` | ||
|
|
||
| ## Behavior When Disabled | ||
|
|
||
| When `verification.enabled` is `false` (default) or `verification` is not configured, the engine skips the gate entirely — existing behavior is unchanged. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haiku pricing example is stale — same issue as in
src/config/types.ts.Line 232 lists
"claude-haiku-4-5" = { inputPer1M = 0.80, outputPer1M = 4.0 }but current Haiku 4.5 pricing is $1 / $5. Since users are likely to copy these examples verbatim, the values should be accurate.📝 Suggested fix
📝 Committable suggestion
🤖 Prompt for AI Agents