[Pelis Agent Factory Advisor] Agentic Workflow Maturity Report — March 2026 #1409
Closed
Replies: 1 comment
-
|
This discussion was automatically closed because it expired on 2026-03-31T03:28:39.574Z.
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
📊 Executive Summary
gh-aw-firewallhas an impressive 21 agentic workflows covering security, CI/CD, documentation, smoke testing, and issue management — placing it firmly in the top tier of agentic workflow adoption. The most significant opportunities are adding issue triage (currently zero automated labeling), a workflow health manager to monitor the growing agent ecosystem, and a container image security scanner to cover the Docker supply chain.🎓 Patterns Learned from Pelis Agent Factory
The Pelis Agent Factory documentation site (https://github.github.io/gh-aw/) describes 100+ workflows used in production across these categories:
How this repo compares: It has implemented the high-value security and CI patterns well, runs three AI engines simultaneously for smoke testing (unique!), and is more security-focused than a typical repo — befitting its domain as a network firewall tool. What it lacks is the meta-observability layer and some of the lower-friction automation patterns (issue triage, code quality agents).
📋 Current Agentic Workflow Inventory
security-guardsecurity-reviewsecret-digger-claudesecret-digger-codexsecret-digger-copilotdependency-security-monitorci-doctorci-cd-gaps-assessmentdoc-maintainercli-flag-consistency-checkertest-coverage-improverissue-monsterissue-duplication-detectorplansmoke-claudesmoke-codexsmoke-copilotsmoke-chrootbuild-testupdate-release-notespelis-agent-factory-advisor🚀 Actionable Recommendations
P0 — Implement Immediately
🏷️ Issue Triage Agent
What: Automatically analyze new issues, apply labels (
bug,security,feature,documentation,question,help-wanted), and leave a brief comment explaining the label and potential next steps.Why: The repo currently has zero automated issue labeling. Issues like
#130(NAT rule bypass) and#422(DNS propagation) sit without labels, making triage invisible. For a security tool with external users, fast triage signals responsiveness and helps maintainers prioritize security bugs.How: Standard issue triage pattern from Pelis Factory — on
issues: [opened, reopened], analyze title/body in context of the codebase (AWF is a network firewall), apply one label from allowed set, comment to explain.Effort: Low (< 1 hour to write, compile, and ship)
🩺 Workflow Health Manager
What: A meta-agent that periodically audits all 21 agentic workflow runs, detecting no-op patterns, excessive costs, misconfigured triggers, and quality degradation. Creates issues for problems found.
Why: Issue
#1401"No-Op Runs" already shows that some workflows are running without producing useful output. With 21 workflows running daily/hourly, invisible inefficiency accumulates quickly. In the Pelis Factory, the Workflow Health Manager created 40 issues, 5 direct PRs + 14 causal chain PRs — one of the highest-leverage workflows. As the workflow collection grows, meta-monitoring becomes essential.How: Use
agentic-workflowstool to analyze recent runs across all workflows. Flag: workflows with zero outputs in past 7 days, workflows with degraded tool calls, unusual token consumption spikes, and failed runs not caught by CI Doctor.Effort: Medium (workflow body requires careful analysis instructions + uses cache-memory for trend tracking)
P1 — Plan for Near-Term
🔍 Breaking Change Checker
What: On every PR, analyze the diff for changes that might break existing users — CLI flag renames/removals, changed default behaviors, removed environment variables, altered Docker API surface.
Why:
gh-aw-firewallis a CLI tool with external users. The repo has open issues about backward compatibility (e.g.,#1328support for common api base) and the AGENTS.md explicitly documents the CLI flag contract. Breaking changes that slip through cause pain at release time. The Pelis Factory breaking change checker created alert issues (e.g., flagging CLI version updates in#14113).How: Trigger on PR, use
bashtool to rungit diff, analyze changes tosrc/cli.ts,action.yml, and container entry points for API-surface changes. Post a review comment if breaking changes detected.Effort: Medium
🐳 Container Image Security Scanner
What: Weekly agentic workflow that triggers
docker scoutortrivyscans on the three container images (squid, agent, api-proxy), analyzes CVE findings, and creates issues for HIGH/CRITICAL vulnerabilities.Why: The firewall's threat model depends on the security of its own container images. The
dependency-security-monitorcovers npm packages, but there's no coverage for the Ubuntu 22.04 base images, squid, or Node.js in api-proxy. A freshubuntu:22.04image can have dozens of unpatched CVEs.How: Weekly schedule, use
bashwithdocker scout cvesortrivy image, parse JSON output, create issues for HIGH+ findings usingcreate-issuesafe output. Deduplicate using cache-memory.Effort: Medium
📊 Audit Workflows (Meta-Agent)
What: Daily workflow that downloads and analyzes logs from all other agentic workflow runs, producing a discussion report covering: success rates, token usage trends, cost estimates, and error patterns.
Why: With 21 workflows, some running hourly, there is no consolidated view of the agent ecosystem's health and cost. The Pelis Factory's Audit Workflows created 93 audit discussions and uncovered optimization opportunities. Given the 3 secret-digger workflows running hourly, visibility into whether they're providing value or just burning tokens is important.
How: Use
agentic-workflowstool to fetch recent runs, aggregate metrics, produce a daily[Audit]discussion with structured tables.Effort: Low-Medium
P2 — Consider for Roadmap
🔧 Code Simplifier
What: Daily agent that analyzes recently modified TypeScript/JavaScript code for complexity, verbosity, and duplication, then creates PRs with targeted simplifications.
Why: The codebase (especially
src/docker-manager.tswhich is 1000+ lines) grows complex over time. The Pelis Factory Code Simplifier achieved 83% merge rate on 6 PRs. With active development adding new features (api-proxy, GHEC support, DLP), a cleanup agent would help maintain readability.How: Daily schedule,
git log --since 7 days, analyze changed TypeScript files, propose simplifications as a draft PR.Effort: Low
🌳 Issue Arborist
What: Periodically analyze open issues to identify related ones, group them as sub-issues under a parent, and create discussion reports about issue relationships.
Why: Issues like
#130(child container NAT bypass),#422(DNS propagation),#240(performance benchmarks),#1328(common api base) are related thematically but not linked. The Pelis Factory Issue Arborist created 18 parent issues to organize work. This would help maintainers see which issues cluster around a theme (e.g., "network isolation improvements").Effort: Low
📝 Schema Consistency Checker
What: Weekly agent that verifies that TypeScript types (
src/types.ts), CLI flags (src/cli.ts),action.ymlinputs,AGENTS.mddocumentation, and environment variable documentation (docs/environment.md) are all consistent with each other.Why: The repo already has the
cli-flag-consistency-checkerfor CLI flags, but broader schema drift between TypeScript interfaces and their documentation is not checked.WrapperConfiginsrc/types.tshas grown to many fields — documentation and code can easily drift.Effort: Low
P3 — Future Ideas
📈 Portfolio Analyst
What: Weekly analysis of the agent portfolio — which workflows are high-ROI (producing PRs/issues that get merged), which are low-ROI (running but not producing value), with token cost estimates.
Why: With 21 workflows and 3 secret-diggers running hourly, cost transparency matters. The Pelis Factory Portfolio Analyst identified "workflows that were costing money unnecessarily."
Effort: Medium
👶 Documentation Noob Tester
What: Monthly agent that reads the docs site from a "first-time user" perspective, follows setup instructions, identifies confusing steps, and creates issues for UX improvements.
Why: The repo has a full Astro/Starlight docs site at
docs-site/. New users setting up AWF for the first time face: Docker requirements, sudo usage, domain whitelisting syntax, and proxy configuration. These are non-trivial.Effort: Medium
🔄 Changeset Generator
What: On PR merge to main, analyze commit messages and generate a changelog entry + version bump suggestion, automating the release prep process.
Why:
update-release-notes.mdalready helps after release, but there's no automation for preparing releases (version bump, changelog). The Pelis Factory Changeset Generator had a 78% merge rate.Effort: Low
📈 Maturity Assessment
Current Overall Level: 4/5 — Above average, with exceptional security coverage (befitting the domain). The primary gap is the observability/meta-monitoring layer and routine code quality automation.
Target Level: 4.5/5 — Adding issue triage (P0) and a Workflow Health Manager (P0) would close the most important gaps with minimal effort.
🔄 Comparison with Best Practices
What it does well:
What it could improve:
Unique domain opportunities:
src/squid-config.ts) against the documented security model to ensure no ACL bypass patterns have crept in📝 Notes for Future Runs
Stored in
/tmp/gh-aw/cache-memory/notes.txt:Beta Was this translation helpful? Give feedback.
All reactions