Agent Persona Exploration - 2026-03-23

[github-actions[bot]]

Persona Overview

• Agent: developer.instructions (agentic-workflows proxy)
• Scenarios Tested: 8 (across 5 personas)
• Average Quality Score: 4.875 / 5.0 ⬆️ (up from 4.77 on 2026-03-21, 4.05 on 2026-02-21)
• Engine Selected: Claude — 100% (8/8 scenarios)

---

Key Findings

• 🏆 5 of 8 scenarios scored a perfect 5.0 — trigger selection is now flawless (avg 5.0/5.0)
• 📦 Pre-processing bash steps are now a consistent pattern for complex log/diff analysis, reducing agent turns and cost
• 🔬 Inline Python (stdlib only) is emerging as the preferred approach for spec/log analysis in strict mode, avoiding pip install entirely
• 💾 cache-memory for persistent state (flaky test quarantine) solves cross-run learning elegantly
• 🔄 Prior issues (excessive doc creation, repo-memory field) appear fully resolved — not observed in any of 8 runs

---

Top Patterns

1. Triggers: path-filtered pull_request (4×), issues: labeled (2×), workflow_run: completed + if:failure (1×), schedule + workflow_dispatch (1×)
2. Tools: GitHub MCP pull_requests/issues/repos (8/8), bash (5/8), cache-memory (1/8 explicit)
3. Security: Read-only permissions + safe-outputs for all writes (8/8); strict:false used once with clear justification (issues:write for incident reporter)
4. Anti-patterns now absent: No wildcard permissions, no direct write tokens, no excessive markdown file creation

---

View Perfect Scores — Top 5 Scenarios (5.0/5.0)

BE-2 · Database Issue Auto-Triage (issues: [opened, edited])
Keyword detection → severity classification → labeled comment. Standout: label allowlist in frontmatter prevents rogue labels; hide-older-comments collapses previous triage on re-edits. Full 4-step structured flow.

FE-1 · Bundle Size Diff on JS/CSS PRs (pull_request + paths)
Auto-detects build tool before running. Reports raw + gzip sizes in table form. PASS/FAIL badge at 5KB threshold. concurrency: cancel-in-progress for rapid push sequences. Handles 0-entry-point repos gracefully.

QA-1 · Flaky Test Detector from CI Logs (workflow_run: CI failure)
10-step bash pre-processing collects logs, parses 4 test output formats (Go, JUnit XML, pytest, generic), fetches 30 historical runs, cross-references failure frequency. Agent then classifies LIKELY_FLAKY / GENUINE_FAILURE. Persistent cache-memory quarantine list survives across branches.

QA-2 · Epic Test Plan Draft (issues: labeled "epic")
Posts 7-section test plan (objectives, scope, happy paths, edge cases, risks, NFRs, open questions). rate-limit: max:5, window:60 guards against bulk-label events. checkout: false speeds startup. Graceful noop for non-software epics.

BE-1 · OpenAPI Breaking Change Detector (pull_request paths: api/**)
Inline Python (stdlib json only) detects 5 categories of breaking changes. Posts up to 15 inline diff comments + REQUEST_CHANGES or COMMENT. Provides 4 migration strategies (versioning, deprecation, make-optional, accept-both) with copy-paste examples.

View Areas for Improvement (Scores 4.4–4.8)

PM-2 · PR TL;DR Summary — 4.4/5.0 (pull_request: labeled "ready-for-review")

• ✅ Label correctly compiled to an if: condition (no wasted runs for other labels)
• ✅ Handles empty PRs gracefully with partial summary
• ⚠️ Does not explicitly enumerate the review threads toolset alongside general comments — the agent may miss structured review thread data
• ⚠️ Prompt could be stronger on distinguishing resolved vs unresolved review threads

DO-1 · Workflow Failure Incident Reporter — 4.8/5.0 (workflow_run: completed)

• ✅ Anti-recursion self-skip prevents the reporter from filing issues about itself
• ✅ Oncall = last human committer is clever and dependency-free
• ⚠️ strict: false required because issues: write conflicts with strict mode — this is a known system limitation, not an agent error, but worth tracking
• ⚠️ Pre-step uses github.event expressions not on the compiler's allowed list (resolved via API calls inside script, but adds complexity)

PM-1 · Weekly Feature Digest — 4.8/5.0 (schedule: monday 09:00)

• ✅ close-older-discussions: true + close-older-key prevents digest accumulation
• ✅ expires: 7d auto-cleanup keeps the Releases category tidy
• ⚠️ References shared/reporting.md (possibly an internal helper that may not exist in all repos)
• ⚠️ Tool selection (4/5): discussions toolset listed but create_discussion goes through safe-outputs — minor redundancy

---

Recommendations

1. Document the pre-processing bash step pattern — Add to AGENTS.md or a skill: for log-heavy workflows (workflow_run, CI analysis), use a bash pre-step to extract/download data before the agent runs. This dramatically reduces turn count and cost.

2. Add rate-limit guidance for issue-label triggers — The rate-limit: max:5, window:60 pattern (seen in QA-2) prevents bulk-label storms from spawning many concurrent agent runs. This should be standard advice for all issues: labeled and pull_request: labeled workflows.

3. Clarify review_comments vs comments toolsets for PR summarization — PM-2 scored lower because the agent didn't explicitly distinguish inline review threads from general comments. Adding a note to the instructions about the get_review_comments vs get_comments distinction would improve PR analysis workflows.

---

Score Trend

Date	Avg Score	Notes
2026-02-21	4.05	First run; doc verbosity issues
2026-03-21	4.77	Improved; 100% Claude first observed
2026-03-23	4.875	Pre-processing pattern; inline Python; persistent cache

References: §23417375175

AI generated by Agent Persona Explorer · history

[pelikhan]

/q the agent personna is meant to update the files in .github/aw/*.md. it is NOT meants to use or consider the AGENTS.md in this project.

[github-actions[bot]]

🔧 Pay attention, 007! Q is preparing your gadgets for this discussion comment...

[github-actions[bot]]

This discussion has been marked as outdated by Agent Persona Explorer.

A newer discussion is available at Discussion #22584.