Daily Perf Improver - Research and Plan

[github-actions[bot]]

Daily Perf Improver - Research and Plan

Performance Landscape Analysis

This document summarizes the current performance state of githubnext/gh-aw and provides a prioritized plan for systematic performance improvements.

Current Performance Baseline

Build & Development Workflow:

• ✅ Build time: 0.125s (target: <2s) - Excellent
• ✅ Full workflow compilation: 2.1s for 74 workflows (target: <10s) - Good
• ⚠️ Unit tests: ~25s (target: <20s) - Needs improvement
• ⚠️ Full tests: ~30s (target: <30s) - At target
• ⚠️ agent-finish: 10-15s (target: <8s) - Needs optimization

CI/CD Performance:

• Test job: 3-5 minutes
• Build job: 5-7 minutes
• JS tests: 2-3 minutes
• Lint job: 3-4 minutes
• Total: 8-12 minutes (acceptable, jobs run in parallel)

Workflow Execution Performance:

• Average tokens per run: ~481k (target: <200k)
• Average cost per run: ~$0.30 (target: <$0.15)
• Execution time: 2-5 minutes for typical workflows

Identified Optimization Targets

Priority 1: High-Impact Development Workflow Optimizations

1. Test Suite Performance (25s → <20s target)

   • Current bottleneck: Sequential test execution
   • Opportunity: Identify and parallelize slow tests
   • Impact: 20-30% faster development cycle
   • Measurement: make test-perf to identify slowest tests

2. Workflow Compilation Incremental Build (2.1s for all → <1s for typical changes)

   • Current bottleneck: All-or-nothing compilation rebuilds all 74 workflows
   • Opportunity: Implement incremental compilation (only rebuild changed .md files)
   • Impact: 50-70% faster recompilation during development
   • Measurement: time ./gh-aw compile before/after changes

3. agent-finish Performance (10-15s → <8s target)

   • Current bottleneck: Sequential execution of build, test, recompile, fmt, lint
   • Opportunity: Parallelize independent operations
   • Impact: 30-40% faster pre-commit validation
   • Measurement: time make agent-finish

Priority 2: CLI Performance Optimizations

4. CLI Startup Time Optimization

   • Current state: Unknown baseline (needs measurement)
   • Opportunity: Lazy loading of schemas and resources
   • Impact: Faster command response times for all CLI operations
   • Measurement: time ./gh-aw --version, time ./gh-aw compile --help

5. MCP Inspection Performance (<2s target)

   • Current state: Needs baseline measurement
   • Opportunity: Caching and efficient parsing
   • Measurement: time ./gh-aw mcp list, time ./gh-aw mcp inspect (workflow)

Priority 3: Workflow Execution Cost Reduction

6. Token Usage Optimization (481k → <200k target)

   • Current bottleneck: Large API responses, especially pull_request_read
   • Opportunity: Use sanitized context text instead of full API responses
   • Impact: 50-60% cost reduction per workflow run
   • Measurement: gh aw logs (workflow) -c 10 with token analysis

7. Tool Call Efficiency (30% reduction target)

   • Current bottleneck: Redundant API calls
   • Opportunity: Better prompt engineering, result caching
   • Impact: 20-30% cost reduction and faster execution
   • Measurement: Analyze tool calls in workflow logs

Performance Engineering Infrastructure

✅ Completed:

• Build steps configuration (.github/actions/daily-perf-improver/build-steps/action.yml)
• Performance guides created:
  • Build performance guide
  • CLI performance guide
  • CI/CD performance guide
  • Workflow performance guide
• Performance testing directories created
• make test-perf command for identifying slow tests

📋 Available Tools:

• time make build - Measure build time
• time make test-unit - Measure unit test performance
• time ./gh-aw compile - Measure compilation time
• make test-perf - Identify slowest 10 tests
• gh aw logs - Analyze workflow execution performance
• gh aw audit (run-id) - Deep performance investigation

Recommended Implementation Order

Week 1-2: Quick Wins (Priority 1)

1. Test suite parallelization - Identify and parallelize slow tests
2. Measure and optimize agent-finish execution

Week 3-4: Compilation Performance (Priority 1)
3. Implement incremental workflow compilation
4. Add compilation cache based on file hashes

Week 5-6: CLI Optimization (Priority 2)
5. Lazy loading of schemas and resources
6. CLI startup time profiling and optimization

Week 7-8: Cost Reduction (Priority 3)
7. Workflow prompt optimization for token efficiency
8. Tool call efficiency improvements

Success Metrics

Development Velocity:

• Unit tests: 25s → <20s (20% improvement)
• agent-finish: 10-15s → <8s (30-40% improvement)
• Recompilation: 2.1s → <1s for typical changes (50%+ improvement)

Cost Efficiency:

• Average tokens: 481k → <200k (58% reduction)
• Average cost: $0.30 → <$0.15 (50% reduction)

Developer Experience:

• Faster feedback loops
• Reduced CI wait times
• Lower workflow execution costs

Next Steps

This plan will be executed incrementally, with each improvement:

1. Measured with before/after benchmarks
2. Validated with existing test suite
3. Documented with reproducible performance testing
4. Submitted as a pull request with evidence of impact

Maintainer feedback welcome on prioritization and approach!

AI generated by Daily Perf Improver

[github-actions[bot]]

Performance Analysis Update - Test Suite Already Optimized

Investigated Priority 1 goal: "Test Suite Performance (25s → <20s target)".

Key Findings

Current Performance (measured 2025-11-12):

• Unit tests (fresh, no cache): 12.0s
• Unit tests (with cache): 5.8s
• Original baseline assumption: 25s

Conclusion: Test suite is already 52% faster than the original baseline assumption (12s vs 25s). The 25s baseline appears to have been from older measurements or different test configurations.

Parallelization Investigation

Tested TestMCPAddIntegration_AddAllServers (2.0s, slowest test) for parallelization:

• Issue discovered: Test uses parent-aggregates-results pattern where parent test checks counters immediately after loop
• Root cause: With t.Parallel(), subtests pause and resume after parent function returns, but parent's defer cleanup() runs before subtests execute
• Learning: Tests with result aggregation in parent aren't suitable for simple t.Parallel() conversion

Alternative approach attempted: Added -parallel 8 flag to test execution

• Result: 12.0s → 11.8s (only 2% improvement)
• Reason: Most tests don't use t.Parallel(), so the flag has minimal effect

Recommendations

1. Test suite goal achieved - Current 12s performance already exceeds <20s target by 40%
2. Focus shift needed - Recommend focusing on other Priority 1 goals:
   • Incremental workflow compilation (2.1s → <1s for single changes)
   • agent-finish optimization (10-15s → <8s)
3. Update baseline - Performance plan should reflect actual 12s baseline (not 25s)

Updated Performance Guide

Added key learnings to .github/copilot/instructions/build-performance.md:

• Document the parent-aggregates-results anti-pattern for parallelization
• Note that -parallel flag only helps tests using t.Parallel()
• Highlight when test patterns aren't suitable for parallelization

Related Items

• Pull Request: #3682

AI generated by Daily Perf Improver

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/daily-perf-improver.md@1f181b37d3fe5862ab590648f25a292e345b5de6. See usage guide.

[github-actions[bot]]

Working on Priority 1 performance improvement from plan: Test suite performance optimization (25s → <20s target).

Initial analysis found dependabot validation tests as slowest (6.95s each out of ~30s total test time). Will investigate and optimize.

Related Items

• Pull Request: #3788

AI generated by Daily Perf Improver

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/daily-perf-improver.md@1f181b37d3fe5862ab590648f25a292e345b5de6. See usage guide.

[github-actions[bot]]

Completed incremental workflow compilation optimization. PR created with 99% performance improvement for unchanged workflows.

Goal: Incremental workflow compilation (2.1s → <1s for single file changes)

Actual Results:

• No changes: 8.2s → 0.05s (99.4% faster)
• Single file change: 8.2s → 0.23s (97% faster)
• Full recompilation: 8.0s (--force-all flag)

Implementation: Hash-based caching system (.gh-aw-cache) that tracks workflow file content SHA256 hashes and skips recompilation of unchanged files.

Next priority from plan: agent-finish optimization (10-15s → <8s target).

Related Items

• Pull Request: #3916

AI generated by Daily Perf Improver

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/daily-perf-improver.md@1f181b37d3fe5862ab590648f25a292e345b5de6. See usage guide.

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.