Skip to content

perf: optimize doc-maintainer token usage (~27% reduction)#1645

Merged
lpcox merged 1 commit intomainfrom
perf/doc-maintainer-token-optimization
Apr 3, 2026
Merged

perf: optimize doc-maintainer token usage (~27% reduction)#1645
lpcox merged 1 commit intomainfrom
perf/doc-maintainer-token-optimization

Conversation

@lpcox
Copy link
Copy Markdown
Collaborator

@lpcox lpcox commented Apr 3, 2026

Problem

The doc-maintainer workflow costs ~$3.91/run and uses ~3,003K tokens. Analysis from the token optimization advisor (#1643) identified that:

  • 22 GitHub MCP tools are loaded but never called — the agent uses bash for all git/file operations
  • The agent spends 3-5 turns on discovery (running git log, find) that can be pre-computed

Changes

1. Remove unused github: toolset (−14% tokens)

The agent never calls any GitHub MCP tools. It uses bash (git/find/cat) for discovery and safeoutputs.create_pull_request for the PR. Each tool schema adds ~600 tokens to every turn's system prompt.

Before: 22 GitHub MCP tools loaded → ~13,200 tokens/turn × 32 turns = ~422K tokens/run
After: 0 GitHub MCP tools loaded

2. Pre-compute git changes in steps: block (−8% tokens)

Added two pre-agent steps:

  • git-changes: Runs git log --since="7 days ago" --name-only and injects results into the prompt via ${{ steps.git-changes.outputs.RECENT_COMMITS }}
  • doc-files: Lists all docs via find docs/ -name "*.md" and injects via ${{ steps.doc-files.outputs.DOC_FILES }}

This eliminates the agent's discovery phase and lets it jump straight to analyzing diffs.

3. max-turns guard (not implemented)

The copilot engine does not support max-turns. The existing timeout-minutes: 15 serves as the cost guard.

Expected Impact

Metric Before After Savings
Total tokens/run 3,003K ~2,200K −27%
Cost/run $3.91 ~$2.55 −35%
GitHub tools loaded 22 0 −100%

Closes #1643

Implements recommendations from token optimization issue #1643:

1. Remove unused github: toolset — the agent uses bash for all git/file
   ops and safeoutputs for PR creation. The 22 GitHub MCP tools were
   loaded but never called, adding ~13,200 tokens/turn to the system
   prompt. Estimated savings: ~422K tokens/run (~14%).

2. Pre-compute git changes and doc file list in steps: block — eliminates
   the agent's discovery phase (3-5 turns of running git log, find).
   Step outputs are injected into the prompt via template variables.
   Estimated savings: ~190K tokens/run (~8%).

3. max-turns guard — not implemented because the copilot engine does not
   support this feature. The existing timeout-minutes: 15 serves as the
   cost cap.

Expected impact: ~$3.91 → ~$2.55/run (-35% cost), 3,003K → ~2,200K
tokens/run (-27%), 22 → 0 GitHub tools loaded (-100%).

Closes #1643

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lpcox lpcox requested a review from Mossaka as a code owner April 3, 2026 18:37
Copilot AI review requested due to automatic review settings April 3, 2026 18:37
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 86.11% 86.21% 📈 +0.10%
Statements 85.98% 86.08% 📈 +0.10%
Functions 87.41% 87.41% ➡️ +0.00%
Branches 78.50% 78.56% 📈 +0.06%
📁 Per-file Coverage Changes (1 files)
File Lines (Before → After) Statements (Before → After)
src/docker-manager.ts 86.4% → 86.8% (+0.39%) 85.9% → 86.3% (+0.38%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

Smoke Test Results

  • ✅ GitHub MCP: feat(smoke-claude): trim unused tools to reduce token spend ~8% / fix: token analyzers should always run and close older issues
  • ✅ Playwright: github.com title contains "GitHub"
  • ✅ File write: /tmp/gh-aw/agent/smoke-test-claude-23957630045.txt created
  • ✅ Bash: file content verified

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

Smoke Test Results

PR Titles: fix: fast-kill agent container on SIGTERM/SIGINT before full cleanup, feat(smoke-claude): trim unused tools to reduce token spend ~8%

  • GitHub MCP (last 2 merged PRs): ✅
  • safeinputs-gh PR query: ❌
  • Playwright github.com title contains GitHub: ❌
  • Tavily search result exists: ❌
  • File write /tmp/gh-aw/agent/smoke-test-codex-23957630051.txt: ✅
  • Bash cat file verification: ✅
  • Discussion query + mystical comment: ❌
  • npm ci && npm run build: ✅
    Overall status: FAIL

🔮 The oracle has spoken through Smoke Codex

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

🤖 Smoke Test Results

PR: perf: optimize doc-maintainer token usage (~27% reduction) — @lpcox

Test Result
GitHub MCP connectivity ✅ PR data fetched
GitHub.com HTTP ✅ (pre-step: file confirms pass)
File write/read /tmp/gh-aw/agent/smoke-test-copilot-23957630066.txt verified

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

Smoke Test: GitHub Actions Services Connectivity ✅

Check Result
Redis PING @ host.docker.internal:6379 PONG
PostgreSQL pg_isready @ host.docker.internal:5432 ✅ accepting connections
PostgreSQL SELECT 1 (db: smoketest, user: postgres) ✅ returned 1

All checks passed.

🔌 Service connectivity validated by Smoke Services

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color ok ✅ PASS
Go env ok ✅ PASS
Go uuid ok ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx All passed ✅ PASS
Node.js execa All passed ✅ PASS
Node.js p-limit All passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #1645 ·

@lpcox lpcox merged commit b097547 into main Apr 3, 2026
57 of 59 checks passed
@lpcox lpcox deleted the perf/doc-maintainer-token-optimization branch April 3, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

⚡ Copilot Token Optimization2026-04-03 — Documentation Maintainer

2 participants