perf: exclude browser tools and reduce turns in smoke-copilot by lpcox · Pull Request #1625 · github/gh-aw-firewall

lpcox · 2026-04-02T20:00:58Z

Summary

Implements all recommendations from #1624 (generated by the Copilot Token Optimization Advisor).

Changes

1. Exclude 21 unused Playwright/browser tools (~10,500 tokens/turn)

Adds --excluded-tools injection to postprocess-smoke-workflows.ts for smoke-copilot. The Copilot CLI includes 21 built-in browser_* tools when --allow-all-tools is set, but smoke-copilot never uses any of them.

2. Remove redundant MCP verification call (saves 1 turn)

The prompt previously instructed the agent to call list_pull_requests to "verify MCP connectivity" — but the pre-step already fetches 2 merged PRs via gh pr list, proving MCP works. Updated to treat pre-fetched data as sufficient.

3. Remove redundant bash echo test (saves 1 turn)

Section 4 (echo "bash works") was redundant since bash functionality is already proven by the file write/read test in Section 3. Removed.

4. Drop `repos` toolset (~2,400 tokens/turn)

Changed toolsets: [repos, pull_requests] → toolsets: [pull_requests]. The 4 tools from repos (list_releases, list_tags, search_repositories, search_code) are never called.

Expected Impact

Metric	Before	After	Savings
Tools visible	34	~9	−25 tools
LLM turns	5	3	−2 turns
Total tokens/run	~374K	~280K	−25%
Billable tokens/run	~211K	~147K	−30%
Estimated cost/run	$0.70	~$0.49	−$0.21

Closes #1624

Implements recommendations from #1624: 1. Exclude 21 unused Playwright/browser tools via --excluded-tools in postprocess-smoke-workflows.ts (saves ~10,500 tokens/turn) 2. Remove redundant MCP verification call — pre-step already proves MCP connectivity, no need for agent to call list_pull_requests 3. Remove redundant bash echo test — bash functionality is already proven by the file write/read test in section 3 4. Drop repos toolset — only pull_requests tools are used, removing 4 unused tool schemas (~2,400 tokens/turn) Expected impact: 5→3 LLM turns, ~25-30% token reduction per run. Closes #1624 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-04-02T20:01:54Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	85.81%	85.91%	📈 +0.10%
Statements	85.69%	85.79%	📈 +0.10%
Functions	86.71%	86.71%	➡️ +0.00%
Branches	78.50%	78.55%	📈 +0.05%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	86.1% → 86.5% (+0.40%)	85.6% → 86.0% (+0.39%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

This PR reduces token and turn usage in the smoke-copilot agentic workflow by trimming unused tools (notably Playwright/browser schemas), removing redundant prompt steps, and narrowing the GitHub MCP toolsets to only what the workflow needs.

Changes:

Injects Copilot CLI --excluded-tools for smoke-copilot.lock.yml to hide unused browser_* tools.
Updates smoke-copilot.md to drop the repos toolset and remove the redundant “bash works” step.
Updates the compiled lock workflow to reflect the prompt/toolset changes and pass --excluded-tools in Copilot runs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scripts/ci/postprocess-smoke-workflows.ts	Adds post-processing to inject `--excluded-tools` for smoke-copilot Copilot CLI invocations.
.github/workflows/smoke-copilot.md	Reduces GitHub MCP toolsets and removes redundant prompt steps/calls to cut turns and tokens.
.github/workflows/smoke-copilot.lock.yml	Regenerated workflow with updated toolsets and Copilot CLI args (including `--excluded-tools`).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-02T20:04:55Z

.github/workflows/smoke-copilot.md

 ### 1. GitHub MCP Testing
-The last 2 merged pull requests have been fetched. Verify MCP connectivity by calling `github-list_pull_requests` for ${{ github.repository }} (limit 1, state merged) and confirm data is returned.
+Pre-step result: MCP connectivity confirmed — 2 merged PRs were pre-fetched successfully (see data below). No additional MCP call is required.



The prompt claims “MCP connectivity confirmed” based on the pre-step, but the pre-step fetches PRs via gh pr list (GitHub API), not via the GitHub MCP server/tools. This change removes the only explicit GitHub MCP tool invocation, so the workflow may no longer detect GitHub MCP breakage. Either reintroduce a minimal GitHub MCP call for this section, or rename/reword the section to reflect that only GitHub API connectivity was pre-validated.

See below for a potential fix:

### 1. GitHub PR Data Pre-Fetch Pre-step result: 2 merged PRs were pre-fetched successfully via GitHub API/CLI (see data below). This validates PR data availability for the test, not GitHub MCP connectivity.

Copilot · 2026-04-02T20:04:56Z

scripts/ci/postprocess-smoke-workflows.ts

+  const isCopilotSmoke = workflowPath.includes('smoke-copilot.lock.yml');
+  if (isCopilotSmoke) {
+    const excludedToolsFlag =
+      '--excluded-tools=browser_close,browser_resize,browser_console_messages,' +
+      'browser_handle_dialog,browser_evaluate,browser_file_upload,browser_fill_form,' +
+      'browser_press_key,browser_type,browser_navigate,browser_navigate_back,' +
+      'browser_network_requests,browser_run_code,browser_take_screenshot,' +
+      'browser_snapshot,browser_click,browser_drag,browser_hover,' +
+      'browser_select_option,browser_tabs,browser_wait_for';
+    const allowAllToolsCount = (content.match(/--allow-all-tools/g) || []).length;
+    if (allowAllToolsCount > 0 && !content.includes('--excluded-tools')) {
+      content = content.replace(
+        /--allow-all-tools/g,
+        `--allow-all-tools ${excludedToolsFlag}`
+      );


This injection is not idempotent for updates: once the workflow content contains any --excluded-tools flag, the script won’t ensure the excluded list matches excludedToolsFlag. That means future changes to the excluded set (or a recompile that adds a different --excluded-tools) won’t be normalized by postprocess. Consider checking for the specific expected flag value (or replacing any existing --excluded-tools=... for smoke-copilot) so updates remain reliable.

github-actions · 2026-04-02T20:05:55Z

Smoke Test Results — PASS

✅ GitHub MCP: feat: add daily token optimization advisor workflows #1620 feat: add daily token optimization advisor workflows; perf: reduce smoke-copilot token usage with pre-steps and tool trimming #1613 perf: reduce smoke-copilot token usage with pre-steps and tool trimming
✅ Playwright: github.com title contains "GitHub"
✅ File write: smoke-test-claude-23919398533.txt created
✅ Bash verify: file read back successfully

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude for issue #1625

github-actions · 2026-04-02T20:07:47Z

Smoke Test: GitHub Actions Services Connectivity ✅

Check	Result
Redis PING (`host.docker.internal:6379`)	✅ `PONG`
PostgreSQL ready (`host.docker.internal:5432`)	✅ `accepting connections`
PostgreSQL `SELECT 1` (db: `smoketest`, user: `postgres`)	✅ Returns `1`

All service connectivity checks passed. (redis-cli was unavailable; Redis was verified via nc.)

🔌 Service connectivity validated by Smoke Services

- Restore MCP call in section 1 so the workflow actually tests GitHub MCP connectivity (not just CLI/API) - Make --excluded-tools injection idempotent: strip any existing flag before re-injecting so updates to the excluded list are always applied correctly on re-runs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-04-02T20:13:56Z

Smoke test results (run 23919740353)

✅ GitHub MCP — #1620 feat: add daily token optimization advisor workflows, #1613 perf: reduce smoke-copilot token usage with pre-steps and tool trimming
✅ Playwright — github.com title contains "GitHub"
✅ File write — /tmp/gh-aw/agent/smoke-test-claude-23919740353.txt created and verified
✅ Bash — file read back successfully

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude for issue #1625

github-actions · 2026-04-02T20:14:54Z

🔥 Smoke Test Results

Test	Result
GitHub MCP	✅ `list_pull_requests` returned PR #1620
GitHub.com HTTP	✅ 200
File write/read	✅ `smoke-test-copilot-23919740341.txt` verified

Overall: PASS

PR by @lpcox, reviewer @Mossaka — perf: exclude browser tools and reduce turns in smoke-copilot

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-04-02T20:16:14Z

Smoke test results:

feat: add daily token optimization advisor workflows
perf: reduce smoke-copilot token usage with pre-steps and tool trimming
GitHub MCP merged PR review: ✅
safeinputs-gh PR query: ❌
Playwright title check: ❌
Tavily web search: ❌
File write + bash cat: ✅
Build (npm ci && npm run build): ✅
Overall status: FAIL

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-04-02T20:17:12Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #1625 · ◷

lpcox requested a review from Mossaka as a code owner April 2, 2026 20:00

Copilot AI review requested due to automatic review settings April 2, 2026 20:00

Copilot started reviewing on behalf of lpcox April 2, 2026 20:01 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

github-actions bot added the smoke-claude label Apr 2, 2026

github-actions bot mentioned this pull request Apr 2, 2026

[agentics] No-Op Runs #1622

Closed

github-actions bot added the smoke-copilot label Apr 2, 2026

This comment has been minimized.

Sign in to view

github-actions bot added the smoke-services label Apr 2, 2026

This comment has been minimized.

Sign in to view

github-actions bot added the build-test label Apr 2, 2026

lpcox merged commit 8fb875b into main Apr 2, 2026
57 of 60 checks passed

lpcox deleted the perf/smoke-copilot-browser-tool-exclusion branch April 2, 2026 21:04

github-actions bot mentioned this pull request Apr 2, 2026

fix: fast-kill agent container on SIGTERM/SIGINT before full cleanup #1623

Merged

4 tasks

lpcox mentioned this pull request Apr 2, 2026

feat: Add daily token usage analysis and optimization workflows github/gh-aw#24186

Closed

Copilot AI mentioned this pull request Apr 2, 2026

feat: Add daily token usage analysis and optimization workflows github/gh-aw#24192

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: exclude browser tools and reduce turns in smoke-copilot#1625

perf: exclude browser tools and reduce turns in smoke-copilot#1625
lpcox merged 2 commits intomainfrom
perf/smoke-copilot-browser-tool-exclusion

lpcox commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

This comment has been minimized.

github-actions bot commented Apr 2, 2026

Uh oh!

This comment has been minimized.

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lpcox commented Apr 2, 2026

Summary

Changes

1. Exclude 21 unused Playwright/browser tools (~10,500 tokens/turn)

2. Remove redundant MCP verification call (saves 1 turn)

3. Remove redundant bash echo test (saves 1 turn)

4. Drop repos toolset (~2,400 tokens/turn)

Expected Impact

Uh oh!

github-actions bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

This comment has been minimized.

github-actions bot commented Apr 2, 2026

Smoke Test: GitHub Actions Services Connectivity ✅

Uh oh!

This comment has been minimized.

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

🔥 Smoke Test Results

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

🏗️ Build Test Suite Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

4. Drop `repos` toolset (~2,400 tokens/turn)

github-actions bot commented Apr 2, 2026 •

edited

Loading