[Pelis Agent Factory Advisor] Pelis Agent Factory Advisor: Workflow Maturity Assessment (2026-03-26)

[github-actions[bot]]

📊 Executive Summary

This repository is in the top tier of agentic workflow adoption, with 21 compiled agentic workflows already deployed across security, CI, documentation, and issue management. The standout strength is exceptional security coverage—three parallel secret-digger red-team agents, a PR security guard, daily deep security reviews, and dependency monitoring. The primary gap is a missing issue triage agent for auto-labeling, and no daily static analysis reports (zizmor/poutine/actionlint)—surprising for a security-focused tool that already uses these in its compile pipeline.

---

🎓 Patterns Learned from Pelis Agent Factory

Key Patterns from Documentation Site

The Pelis Agent Factory ran 100+ workflows in the github/gh-aw repository, exploring the philosophy of "let's create a new automated agentic workflow for that" for nearly every opportunity.

Top patterns observed:

Pattern	Description	Example
Continuous Quality	Daily/weekly agents that improve code without human intervention	Code Simplifier (83% merge rate), Duplicate Code Detector (79% merge rate)
Fault Investigation	Agents triggered on failures to auto-diagnose	CI Doctor (69% merge rate, 9 merged PRs)
Meta-Agents	Workflows that watch other workflows	Workflow Health Manager (40 issues, 34 downstream PRs)
ChatOps/Slash Commands	On-demand agents invoked by /command	Q optimizer (78% merge rate, 69 merged PRs)
Causal Chain Automation	Agent A creates issue → Agent B fixes it → high merge rates	Testify Expert: 100% causal chain merge rate
Security Specialization	Multiple focused security agents vs. one monolith	Firewall, Secrets Analysis, Malicious Code Scan, Static Analysis Report all separate
Triplication of Engines	Same task run by different AI engines for coverage	secret-digger-{claude,codex,copilot} pattern already in this repo

Key Patterns from Agentics Repository (githubnext/agentics)

The agentics reference repository has these relevant patterns not yet in this repo:

• daily-test-improver.md - Identifies coverage gaps and writes tests
• ci-coach.md - Analyzes CI pipelines for optimization opportunities
• issue-arborist.md - Links related issues as sub-issues (77 discussion reports, 18 parent issues)
• grumpy-reviewer.md - Opinionated code reviewer with personality
• q.md - On-demand workflow optimizer via slash command
• sub-issue-closer.md - Auto-closes completed sub-issues

Comparison to This Repository

This repo has internalized the security specialization pattern exceptionally well (3 secret diggers, separate security guard + security review + dependency monitor). However, it is under-indexed on continuous code quality and meta-monitoring patterns.

---

📋 Current Agentic Workflow Inventory

Workflow	Purpose	Trigger	Assessment
build-test	Build/test on PRs	PR opened/sync	✅ Well-configured
ci-cd-gaps-assessment	Daily CI/CD gap analysis	Daily	✅ Good coverage
ci-doctor	Investigate CI failures	workflow_run failure	✅ Excellent - follows Pelis pattern
cli-flag-consistency-checker	CLI docs consistency	Weekly	✅ Domain-specific value
dependency-security-monitor	Dep vulnerability monitoring	Daily	✅ Critical for security tool
doc-maintainer	Keep docs in sync with code	Daily	✅ Well-designed
issue-duplication-detector	Detect duplicate issues	issues.opened	✅ Good hygiene
issue-monster	Assign issues to Copilot SWE	issues.opened + hourly	✅ Task dispatcher pattern
pelis-agent-factory-advisor	This workflow	Daily	✅ Meta-advisory
plan	/plan slash command	slash_command	✅ ChatOps pattern
secret-digger-claude	Red-team secrets scan (Claude)	Hourly	✅ Unique - engine triplication
secret-digger-codex	Red-team secrets scan (Codex)	Hourly	✅ Unique - engine triplication
secret-digger-copilot	Red-team secrets scan (Copilot)	Hourly	✅ Unique - engine triplication
security-guard	PR security review	PR opened/sync	✅ Claude-powered, domain-expert
security-review	Deep daily security analysis	Daily	✅ Comprehensive STRIDE modeling
smoke-chroot	Chroot integration smoke test	PR on src/containers + 🚀 reaction	✅ Reaction-triggered pattern
smoke-claude	Claude smoke test	PR + every 12h + ❤️ reaction	✅ Multi-engine smoke coverage
smoke-codex	Codex smoke test	PR + every 12h + 🎉 reaction	✅ Multi-engine smoke coverage
smoke-copilot	Copilot smoke test	PR + every 12h + 👀 reaction	✅ Multi-engine smoke coverage
test-coverage-improver	Add security test coverage	Weekly	✅ Security-focused, good skip-if-match
update-release-notes	Enhance release notes post-publish	release.published	✅ Covers post-release

---

🚀 Actionable Recommendations

P0 - Implement Immediately

[P0] Issue Triage Agent

What: Automatically label new issues with bug, feature, enhancement, documentation, question, or security based on content analysis. Comment on the issue explaining the label and hinting at next steps.

Why: This is the "hello world" of agentic workflows in Pelis. Every issue currently lands unlabeled unless a human reviews it. The issue-monster then has to pick up unlabeled issues without knowing their nature. Adding auto-labeling creates a natural pipeline: triage → issue-monster → Copilot SWE agent.

How: Simple workflow triggered on issues: [opened, reopened] with add-labels and add-comment safe outputs. High priority for security label given the domain.

Effort: Low (30 min to write, already have the safe-output tools)

---
on:
  issues:
    types: [opened, reopened]
permissions:
  issues: read
tools:
  github:
    toolsets: [issues, labels]
safe-outputs:
  add-labels:
    allowed: [bug, feature, enhancement, documentation, question, security, good-first-issue]
  add-comment: {}
timeout-minutes: 5
---
# Issue Triage Agent
Label incoming issues for $\{\{ github.repository }}. For security-focused issues (firewall bypass, credential leaks, Docker escapes), add the `security` label. For each labeled issue, comment briefly explaining the categorization and whether it's suitable for automated fixing.

---

[P0] Daily Static Analysis Report

What: Daily workflow running zizmor, poutine, and actionlint on the compiled lock files and workflows, posting results as a discussion.

Why: This repo already runs these tools in the CI compile step, but only on changed files. A daily full-repo static analysis would catch security drift in workflow files—especially important since this is a security tool whose own workflows should be above reproach. The Pelis factory ran 57 such discussions, making it one of their most active security workflows.

How: Use agenticworkflows-compile with --zizmor --poutine --actionlint flags, analyze results, and post as a [Static Analysis]-prefixed discussion.

Effort: Low (the compile tool already has these flags; workflow body is straightforward)

---

P1 - Plan for Near-Term

[P1] Daily Malicious Code Scan

What: Daily scan of commits from the past 24 hours looking for suspicious patterns: unexpected network endpoints hardcoded, obfuscated strings, unusual base64 blobs, supply chain attack indicators in dependency files.

Why: AWF is a security-critical tool. A sophisticated attacker compromising a PR could insert malicious code designed to phone home or weaken firewall rules. As Pelis notes, "it happens." The repo already runs hourly secret diggers inside the agent container—this would scan the source code itself.

How: Read recent commits via git log --since 24h, analyze diffs for suspicious patterns using bash + LLM reasoning. Post findings as issues tagged [Malicious Code Scan].

Effort: Low-Medium

---

[P1] Changeset / Version Bump Automation

What: When a PR is merged to main with feat: or fix: commits (conventional commits are enforced), automatically open a draft PR that bumps the version in package.json and updates CHANGELOG.md.

Why: The repo has update-release-notes (post-publish) but nothing pre-release. The Pelis Changeset workflow had 78% merge rate (22/28 PRs). This repo enforces conventional commits via commitlint—perfect input signal.

How: Triggered on push: [main], analyzes commits since last tag with git log, determines major/minor/patch bump, updates package.json and changelog, opens draft PR with [chore] bump version prefix.

Effort: Medium

---

[P1] Breaking Change Checker

What: PR-triggered agent that analyzes changes to CLI flags (src/cli.ts), Docker API (src/docker-manager.ts), and public TypeScript interfaces (src/types.ts) to detect backward-incompatible changes and alert maintainers.

Why: AWF is used by CI pipelines and scripts that depend on stable CLI flags and container behavior. Unannounced breaking changes are high-risk. The Pelis Breaking Change Checker creates alert issues (e.g., #14113).

How: On PR open/sync, diff src/cli.ts and src/types.ts against main. Look for removed/renamed flags, changed default values, altered container networking parameters. Create an issue if breaking changes detected.

Effort: Medium

---

P2 - Consider for Roadmap

[P2] Workflow Health Manager

What: A weekly meta-agent that reviews the health of all other agentic workflows: checks for workflows that haven't run recently, identifies ones with high failure rates, spots outdated dependencies in workflow configs.

Why: With 21 workflows, some inevitably go stale, have broken triggers, or drift from the codebase. The Pelis Workflow Health Manager created 40 issues and had 25 lead to 34 PRs. This repo's ci-doctor handles individual failures but not systemic health.

How: Use agenticworkflows-status and agenticworkflows-logs to get run history. Flag workflows that haven't run in 7+ days (possible broken trigger), have >50% failure rate, or aren't in the ci-doctor watchlist.

Effort: Medium

---

[P2] Code Simplifier

What: Daily agent that looks at recently modified TypeScript files (src/*.ts) and proposes simplifications: extracting repeated patterns, simplifying nested conditionals, consolidating error handling.

Why: AWF is a complex codebase with Docker orchestration, iptables rules, and Squid config generation. Complexity accumulates. The Pelis Code Simplifier had 83% merge rate (5/6 PRs). Given the security-critical nature, simpler code is also safer code.

How: git diff --name-only HEAD~5 HEAD to find recently changed files, analyze with LLM, propose simplifications via draft PR. Skip test files and lock files.

Effort: Low-Medium

---

[P2] Issue Arborist

What: Weekly agent that scans open issues and groups related ones (e.g., multiple issues about DNS exfiltration, or several about IPv6 handling) by creating a parent issue and linking children as sub-issues.

Why: As AWF grows, related issues accumulate without structural organization. The Pelis Issue Arborist created 18 parent issues to group related work. Given that AWF has issue-monster dispatching issues to Copilot SWE, organized sub-issues would give it better batched context.

Effort: Medium

---

[P2] Grumpy Reviewer / Q Optimizer

What:

• Grumpy Reviewer: Triggered by /grumpy slash command on a PR—performs opinionated code review with a sharp eye for security regressions specific to AWF (iptables rule order, Squid ACL bypass, capability dropping completeness).
• Q Optimizer: Triggered by /q slash command—investigates workflow issues and proposes fixes.

Why: Security Guard already reviews all PRs for security. Grumpy Reviewer would add a general code quality layer with domain expertise, invoked on-demand. Q would give maintainers an interactive way to debug workflow problems.

Effort: Medium each

---

P3 - Future Ideas

[P3] Weekly Issue Summary

What: Weekly digest posted as a discussion summarizing open issues by category, newly closed issues, and flagging issues that have been stale for 30+ days.

Why: Community visibility and maintainer awareness. Low immediate impact but good hygiene.

---

[P3] Daily Performance Monitor Enhancement

What: The existing performance-monitor.yml (non-agentic) could be complemented with an LLM-powered agent that interprets benchmark regressions and proposes fixes.

Why: Converts raw timing data into actionable insights.

---

[P3] Agentic Wiki / Contribution Guide Agent

What: An agent (from githubnext/agentics/agentic-wiki-coder.md pattern) that maintains a living CONTRIBUTING.md based on actual recent PR activity—updating it when new patterns emerge (e.g., when a new container was added, when the AWF --build-local flag pattern appeared).

Why: CONTRIBUTING.md tends to go stale quickly in active projects.

---

📈 Maturity Assessment

Dimension	Level	Notes
Security automation	⭐⭐⭐⭐⭐ (5/5)	Exceptional: 3 red-team agents, security guard, deep review, dep monitor
CI/CD automation	⭐⭐⭐⭐ (4/5)	Strong: CI doctor, 4 smoke tests, build-test, coverage improver
Documentation automation	⭐⭐⭐ (3/5)	Good: doc-maintainer, CLI checker; missing static analysis reports
Issue management	⭐⭐⭐ (3/5)	Partial: issue-monster, dedup detector, plan; missing triage
Code quality automation	⭐⭐ (2/5)	Gap: no code simplifier, no duplicate detector
Meta-monitoring	⭐⭐ (2/5)	Gap: no workflow health manager
Release automation	⭐⭐⭐ (3/5)	Partial: release notes updater; missing changeset/version bump

Overall Current Level: 4/5 — Highly mature, especially for a security tool. Among the top 10% of repositories in agentic workflow adoption.

Target Level: 4.5/5 — Achievable by implementing the P0 and P1 recommendations above.

Gap to Close: Issue triage, static analysis reports, malicious code scan, and changeset automation would fill the most significant gaps with relatively low effort.

---

🔄 Comparison with Best Practices

What This Repository Does Exceptionally Well

1. Engine diversity for security testing: Running secret-digger with Claude, Codex, AND Copilot is beyond what Pelis did—this is innovative and domain-appropriate for a firewall tool that is itself used with these engines.
2. Reaction-triggered smoke tests: Using emoji reactions (🚀❤️🎉👀) as triggers for engine-specific smoke tests is elegant and low-friction.
3. Domain-specific security guard: The security guard is deeply tailored to AWF's specific attack surface (iptables, Squid ACL, capability dropping)—far more useful than a generic code reviewer.
4. Self-aware CI doctor: The CI doctor explicitly knows about AWF-specific failure patterns (Docker network conflicts, iptables issues).

What Could Improve

1. Missing causal chain automation: The Pelis factory's highest-impact pattern was Agent A creates issue → Agent B (usually issue-monster → Copilot SWE) fixes it. This repo has issue-monster but no upstream agents (like issue triage, code simplifier) creating well-labeled issues for it to consume.
2. Static analysis gap: Despite using zizmor/poutine in the compile pipeline, there's no daily report on workflow security posture—ironic for a security tool.
3. No meta-monitoring: With 21 workflows, there's no agent watching whether workflows are healthy, running, or drifting.

Unique Opportunities Given the Domain

This repository's domain (firewall/security for AI agents) creates unique automation opportunities not seen in the Pelis factory:

• Firewall escape test reports could feed the security-review workflow as a data source (the security-review workflow already looks for this)
• Cross-repo security advisories: An agent could monitor CVE feeds for Docker/Squid/iptables vulnerabilities specifically
• Smoke test result analysis: An agent that compares smoke test outcomes across engines (Claude vs Codex vs Copilot) to detect engine-specific behavioral differences in the firewall

---

📝 Notes for Future Runs

Run date: 2026-03-26 | 21 agentic workflows found | First run (no prior state)

Items to track over time:

• Did issue-triage get added? (P0 gap)
• Did static analysis report get added? (P0 gap)
• Are smoke test results being analyzed cross-engine?
• Is issue-monster dispatch rate healthy? (check issues being assigned vs. resolved)
• Workflow health: any workflows going stale?

AI generated by Pelis Agent Factory Advisor

• expires on Apr 2, 2026, 3:37 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-04-02T03:37:36.678Z.

Closed by Workflow