Skip to content

feat(heartbeat): add triage model for cheap heartbeat pre-screening#786

Open
zombopanda wants to merge 8 commits intopaperclipai:masterfrom
zombopanda:feat/heartbeat-triage-model
Open

feat(heartbeat): add triage model for cheap heartbeat pre-screening#786
zombopanda wants to merge 8 commits intopaperclipai:masterfrom
zombopanda:feat/heartbeat-triage-model

Conversation

@zombopanda
Copy link

Summary

  • Adds optional heartbeatModel field to agent adapter config, allowing a cheaper model (e.g. Haiku) to pre-screen timer-triggered heartbeats before launching the full model
  • When a heartbeat fires on timer and heartbeatModel is set, runs a short triage phase (max 3 turns) that checks for assigned work via Paperclip API and returns a structured decision: none (no work), handle (simple work done), or escalate (launch full model)
  • Triage failures always default to "escalate" to ensure work is never silently skipped
  • Non-timer wakeups (on_demand, assignment, automation) skip triage entirely

Motivation

Agents that heartbeat frequently (e.g. every 5 minutes) but only have intermittent work waste significant tokens/cost on the full model just to discover there's nothing to do. A cheap triage model can check for work in 3 turns at a fraction of the cost, only escalating to the full model when real work exists.

Changes

File Change
packages/adapter-utils/src/types.ts Add heartbeatModel to CreateConfigValues
server/src/services/heartbeat.ts Add TRIAGE_SYSTEM_PROMPT, parseTriageOutput(), triage phase in executeRun(), triage usage merging
packages/adapters/claude-local/src/index.ts Document heartbeatModel in agentConfigurationDoc
ui/src/components/AgentConfigForm.tsx Add HeartbeatTriageModelDropdown component, shown when heartbeat is enabled
ui/src/components/agent-config-defaults.ts Add heartbeatModel: "" default
server/src/__tests__/heartbeat-triage.test.ts 18 tests covering parse logic, triage conditions, and safety guarantees

Usage

  1. Go to agent config -> Run Policy
  2. Enable heartbeat
  3. Select a "Heartbeat triage model" (e.g. Claude Haiku)
  4. Timer heartbeats now run triage first; on-demand/assignment wakes skip triage

Test plan

  • parseTriageOutput correctly parses all action types from stdout
  • Unparseable/malformed output defaults to "escalate"
  • Triage skipped when heartbeatModel is not set
  • Triage skipped for non-timer sources (on_demand, assignment, automation)
  • Triage cost tracked in run's usageJson
  • TypeScript compiles cleanly across all packages
  • Manual: configure agent with heartbeatModel, verify timer heartbeats triage first
  • Manual: verify on-demand wake skips triage

🤖 Generated with Claude Code

When an agent has `heartbeatModel` configured in its adapterConfig, timer-
triggered heartbeats run a short triage phase first using the cheaper model.
The triage model checks for assigned work via the Paperclip API and returns
a structured JSON decision: "none" (no work, exit early), "handle" (simple
work handled by triage), or "escalate" (complex work, launch full model).

This significantly reduces costs for agents that heartbeat frequently but
only have work intermittently. Non-timer sources (on_demand, assignment,
automation) skip triage entirely and run the full model immediately.

Triage failures always default to "escalate" to ensure work is never
silently skipped.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 13, 2026

Greptile Summary

This PR introduces a triage pre-screening phase for timer-triggered heartbeats: a cheap model (e.g. Haiku) runs up to 3 turns to check for assigned work and returns a structured none / handle / escalate decision before the full model is launched, reducing token spend on idle heartbeats. The brace-balanced JSON parser, safety-first escalation defaults, and triage-phase isolation (own runId suffix, separate log prefix, cancellation guard) are all well-considered. However, several issues remain after the previous review round:

  • workspaceStrategy inherited by triage config (server/src/services/heartbeat.ts): the triage run will execute a full git-worktree setup/teardown for every timer heartbeat despite the triage prompt being purely API-based, potentially adding more latency and cost than the model savings offset. Should be explicitly set to undefined alongside instructionsFilePath.
  • Inconsistent usageJson schema (server/src/services/heartbeat.ts): triage-only runs (early exit) spread triage token counts at the top level, while triage+escalate runs nest them under a triageUsage key. Any analytics or billing consumer reading usageJson.costUsd will produce incorrect totals depending on which code path ran.
  • Overlay form clears heartbeatModel to undefined instead of "" (ui/src/components/AgentConfigForm.tsx): v || undefined when v === "" records undefined in the overlay, which typically means "inherit from base config" — so clearing the dropdown in an overlay could silently re-enable a triage model set in the underlying config.

Confidence Score: 2/5

  • Not safe to merge yet — the usageJson schema inconsistency will silently corrupt cost tracking, and the inherited workspaceStrategy undermines the core cost-saving rationale.
  • The safety-first escalation guarantee is solid, but three new issues were found on top of the unresolved previous-thread concerns: inconsistent usageJson structure breaks analytics consumers, inherited workspaceStrategy adds full workspace overhead to every triage run, and the overlay form's v || undefined pattern can re-inherit a triage model the user explicitly cleared.
  • server/src/services/heartbeat.ts requires the most attention — both the workspaceStrategy inheritance and the usageJson schema inconsistency are in this file. ui/src/components/AgentConfigForm.tsx needs the overlay onChange fix.

Important Files Changed

Filename Overview
server/src/services/heartbeat.ts Core triage implementation — contains three issues: workspaceStrategy inherited (adds workspace overhead to each triage run), inconsistent usageJson schema between triage-only and escalated runs, and prior-thread issues around session state, cancellation race, and stdout contamination.
ui/src/components/AgentConfigForm.tsx Adds HeartbeatTriageModelDropdown component; overlay form path passes undefined instead of "" when clearing the model, which may cause triage model to be inherited from base config rather than explicitly disabled.
server/src/tests/heartbeat-triage.test.ts 18 tests covering parse logic, action routing, and safety guarantees; integration-logic tests are thin (testing pure JS expressions rather than actual service behaviour), but parse tests are thorough and cover brace-in-reason edge cases.
packages/adapter-utils/src/types.ts Adds required heartbeatModel: string field to CreateConfigValues; consistent with the existing non-optional boolean/number fields in the same interface.
ui/src/components/agent-config-defaults.ts Adds heartbeatModel: "" default — correct empty-string sentinel that disables triage by default.
packages/adapters/claude-local/src/index.ts Documentation-only change adding heartbeatModel to agentConfigurationDoc; description is accurate.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: server/src/services/heartbeat.ts
Line: 1533-1541

Comment:
**`workspaceStrategy` inherited from full agent config**

`triageConfig` spreads `resolvedConfig` and explicitly nulls out `bootstrapPrompt` and `instructionsFilePath`, but does **not** clear `workspaceStrategy`. If the agent has a `workspaceStrategy` (e.g. `{ type: "git_worktree", ... }`), every triage run will go through a full git worktree setup and teardown — even though the triage prompt only reads from the Paperclip API and never touches the filesystem. Worktree creation can take several seconds, directly undermining the cost-saving motivation for triage.

Consider explicitly clearing it alongside the other lightweight overrides:

```typescript
const triageConfig = {
  ...resolvedConfig,
  model: heartbeatModel,
  maxTurnsPerRun: TRIAGE_MAX_TURNS,
  promptTemplate: TRIAGE_PROMPT_TEMPLATE,
  bootstrapPrompt: "",
  instructionsFilePath: undefined,
  workspaceStrategy: undefined, // triage is API-only, no workspace needed
  timeoutSec: 120,
};
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: server/src/services/heartbeat.ts
Line: 1579-1581

Comment:
**Inconsistent `usageJson` schema between triage-only and triage+escalate runs**

The two paths produce structurally different `usageJson` objects:

- **Triage-only early exit** (`"none"` / `"handle"`): triage tokens are promoted to the *top level* with a `triageOnly: true` marker:
  ```json
  { "inputTokens": 100, "outputTokens": 50, "costUsd": 0.001, "triageOnly": true }
  ```

- **Triage + escalate path** (lines ~1750–1754): full-model tokens are top-level and triage is *nested* under `triageUsage`:
  ```json
  { "inputTokens": 5000, "outputTokens": 2000, "costUsd": 0.05, "triageUsage": { ... } }
  ```

Any analytics or billing code that reads `usageJson.inputTokens` / `usageJson.costUsd` will silently produce wrong cost summaries depending on which path ran. A consistent structure — e.g., always storing main-model usage at the top level with a nested `triageUsage` key (and `null` main-model usage for triage-only runs) — would let consumers handle both cases uniformly.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: ui/src/components/AgentConfigForm.tsx
Line: 836

Comment:
**Clearing triage model in overlay sends `undefined` instead of `""`**

```tsx
onChange={(v) => mark("heartbeat", "heartbeatModel", v || undefined)}
```

When the user selects "None (skip triage)", `HeartbeatTriageModelDropdown` calls `onChange("")`. Here `"" || undefined` evaluates to `undefined`, so `mark` records the field as `undefined` — which in the overlay system typically means "inherit from base config" rather than "explicitly unset". If the base agent config already has a `heartbeatModel` set, clearing it in an overlay would silently re-inherit the base value instead of disabling triage.

Compare with the direct-edit path (line 807) where `onChange={(v) => set!({ heartbeatModel: v })}` correctly stores `""`.

```suggestion
              onChange={(v) => mark("heartbeat", "heartbeatModel", v === "" ? "" : v)}
```

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: bd47f9f

Claude and others added 2 commits March 13, 2026 14:54
The previous commit only added heartbeatModel documentation to
claude-local. This adds the same heartbeat triage section to the
remaining 6 adapters: codex-local, cursor-local, gemini-local,
openclaw-gateway, opencode-local, and pi-local.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CRITICAL: pass TRIAGE_SYSTEM_PROMPT via promptTemplate (not dead
  context.systemPrompt that no adapter reads)
- Replace fragile regex parser with brace-balanced JSON extractor
  that handles } inside reason strings
- Fix parser to skip past non-matching JSON objects instead of
  re-scanning their contents
- Remove dead TriageResult.usage field (actual usage flows through
  separate triageUsage local)
- Preserve sessionIdAfter on triage early-exit path
- Remove unused _paperclipTriageMode context key
- Add tests: braces in reason, multiple JSON objects in stdout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zombopanda
Copy link
Author

@greptile-apps, fixed

…config, sort model list

- Use `${runId}_triage` to prevent per-run state collisions between triage and full-model adapter calls
- Null out bootstrapPrompt and instructionsFilePath in triage config to avoid inheriting full agent startup instructions and reading CLAUDE.md on every timer heartbeat
- Sort triage model dropdown: lighter models (Haiku) first, show warning hint when expensive model selected

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zombopanda
Copy link
Author

@greptile-apps, fixed

- On triage "handle" early-exit, call updateRuntimeState and upsertTaskSession
  so next heartbeat sees the triage run instead of stale session data
- Prefix triage log output with [triage] to distinguish from full-model output
  in stdoutExcerpt/stderrExcerpt

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zombopanda
Copy link
Author

@greptile-apps, fixed again

Check run status before overwriting with "succeeded" — if the run was
cancelled by the control plane during triage execution, respect the
cancellation instead of silently overwriting it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zombopanda
Copy link
Author

@greptile-apps, fixed again

…e timeout

- Rename constant to match actual usage (promptTemplate, not system prompt)
- Add timeoutSec: 120 to triage config — triage should complete in seconds,
  not inherit the full-model timeout of tens of minutes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zombopanda
Copy link
Author

@greptile-apps, fixed again

- Clear workspaceStrategy in triageConfig so triage runs skip
  unnecessary git worktree setup/teardown
- Make triage-only usageJson consistent with escalate path: always
  nest triage tokens under triageUsage key with null main-model fields

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zombopanda
Copy link
Author

@greptile-apps, fixed again — stripped workspaceStrategy from triage config and unified usageJson schema across both paths.

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 13, 2026

Nice! The schema is now consistent:

  • Triage-only: { inputTokens: null, outputTokens: null, costUsd: null, triageUsage: {...} }
  • Escalate: { inputTokens: X, outputTokens: Y, costUsd: Z, triageUsage: {...} }

And workspaceStrategy: undefined makes sense since triage is just API calls. LGTM! ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant