UX Analysis Report — 2026-03-31

[github-actions[bot]]

Targeted enterprise UX analysis covering 1 documentation guide, 1 CLI command, 2 workflow message configurations, and 1 validation file.

Executive Summary

Today's analysis focused on:

• 2 documentation file(s): getting-started-mcp.md, faq.md
• 1 CLI command: validate
• 2 workflow message configurations: ci-doctor.md, archie.md
• 1 validation file: pkg/workflow/agent_validation.go

Overall Quality: Professional — the codebase demonstrates strong UX discipline. Help text is comprehensive, documentation is well-structured, and error messages are actionable.

Key Finding: The Getting Started with MCP guide introduces mode: remote without warning users that it is incompatible with the default GITHUB_TOKEN, causing silent authentication failures for users following the tutorial path.

---

Quality Highlights ✅

Example 1: validate CLI Command Help Text

• File: pkg/cli/validate_command.go
• What works well: Clear short description, excellent coverage of 7+ distinct usage examples, explains equivalence to compile flags, includes the WorkflowIDExplanation reusable block
• Rating: ✅ Professional

Example 2: docs/src/content/docs/reference/faq.md

• What works well: Logical section groupings (Determinism → Capabilities → Security), practical tone that respects user expertise, good use of cross-links to deeper reference material, accurately handles nuanced scenarios like cross-repository access
• Rating: ✅ Professional

---

Improvement Opportunities 💡

High Priority

Opportunity 1: Missing Authentication Warning for Remote Mode — Single File

• File: docs/src/content/docs/guides/getting-started-mcp.md
• Current State (line 104):

  "Remote mode (mode: remote) connects to a hosted server for faster startup with no Docker required."

• Issue: The description highlights the benefit (no Docker) but omits the critical constraint: mode: remote requires a PAT or GitHub App token. It does not work with the default GITHUB_TOKEN used in all examples in the same guide.
• User Impact: A developer following this Getting Started tutorial who adds mode: remote to try the "faster startup" option will receive confusing authentication errors with no indication of the cause. The fix requires navigating to a separate reference page. High-frequency path — this is the main onboarding guide for MCP.
• Suggested Change: Add a concise constraint note to the Operating Modes description
• Design Principle: Clarity and Precision — explicit expectations reduce debugging time

Medium Priority

Opportunity 2: Non-Actionable Engine Feature Error Messages — Single File

• File: pkg/workflow/agent_validation.go
• Current State (lines 127, 150):

  "max-turns not supported: engine '%s' does not support the max-turns feature"
"max-continuations not supported: engine '%s' does not support the max-continuations feature"

• Issue: The messages correctly identify the problem but do not tell the user which engines do support these features, leaving them without a clear resolution path. The "not supported" prefix and "does not support" suffix also create redundancy.
• User Impact: Users who configure max-turns on an unsupported engine receive an error and must search documentation to find compatible engines. An actionable message would eliminate this lookup.
• Design Principle: Trust and Reliability — error messages should provide actionable solutions

---

Files Reviewed

Documentation

• docs/src/content/docs/guides/getting-started-mcp.md — Rating: ⚠️ Needs Minor Work
• docs/src/content/docs/reference/faq.md — Rating: ✅ Professional

CLI Commands

• validate (pkg/cli/validate_command.go) — Rating: ✅ Professional

Workflow Messages

• .github/workflows/ci-doctor.md — Rating: ✅ Professional (persona branding is intentional)
• .github/workflows/archie.md — Rating: ✅ Professional

Validation Code

• pkg/workflow/agent_validation.go — Rating: ⚠️ Needs Minor Work

---

Metrics

• Files Analyzed: 6
• Quality Distribution:
  • ✅ Professional: 4
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

---

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

---

Task 1: Add Authentication Warning to Remote Mode — getting-started-mcp.md

File to Modify: docs/src/content/docs/guides/getting-started-mcp.md

Current Experience

Lines 102–106 describe operating modes. The remote mode description presents it as a straightforward performance option without surfacing a key constraint:

### Operating Modes

Remote mode (`mode: remote`) connects to a hosted server for faster startup with no Docker required. Local mode (`mode: local`) runs in Docker, enabling version pinning for offline or restricted environments. See [Remote vs Local Mode](/gh-aw/reference/github-tools/#github-tools-remote-mode).

Quality Issue

Design Principle: Clarity and Precision

This is a Getting Started guide — readers are actively setting up their first workflow. The omission of the PAT requirement means users may attempt to use mode: remote with GITHUB_TOKEN and receive cryptic authentication failures. The fix is in the reference docs but not surfaced here. The cost of the confusion is high; the cost of one sentence is zero.

Proposed Improvement

Before:

Remote mode (`mode: remote`) connects to a hosted server for faster startup with no Docker required. Local mode (`mode: local`) runs in Docker, enabling version pinning for offline or restricted environments. See [Remote vs Local Mode](/gh-aw/reference/github-tools/#github-tools-remote-mode).

After:

Remote mode (`mode: remote`) connects to a hosted server for faster startup with no Docker required. **Note**: Remote mode requires a PAT or GitHub App token — it is not compatible with the default `GITHUB_TOKEN`. Local mode (`mode: local`, the default) runs in Docker and works with standard workflow tokens. See [Remote vs Local Mode](/gh-aw/reference/github-tools/#github-tools-remote-mode).
```

**Why This Matters**
- **User Impact**: Prevents silent authentication failures for users following the Getting Started tutorial path
- **Quality Factor**: Explicit expectations — the constraint is mentioned at the point of introduction
- **Frequency**: The Getting Started guide is the highest-traffic onboarding path for new MCP users

**Success Criteria**
- [ ] Changes made to `docs/src/content/docs/guides/getting-started-mcp.md` only
- [ ] Remote mode section accurately communicates the PAT requirement
- [ ] Quality rating improves from ⚠️ to ✅

**Scope Constraint**
- **Single file only**: `docs/src/content/docs/guides/getting-started-mcp.md`
- No changes to other files required
- Can be completed independently

---

#### Task 2: Improve Engine Feature Error Messages — `agent_validation.go`

**File to Modify**: `pkg/workflow/agent_validation.go`

**Current Experience**

Lines 127 and 150 emit these errors when an unsupported feature is used:
```
max-turns not supported: engine 'codex' does not support the max-turns feature
max-continuations not supported: engine 'copilot' does not support the max-continuations feature

Quality Issue

Design Principle: Trust and Reliability

The messages identify the problem but provide no solution. An enterprise user hitting this error must open documentation to discover which engines support these features. A better message includes the supported engines inline, enabling immediate self-service resolution.

Proposed Improvement

Before (line 127):

return fmt.Errorf("max-turns not supported: engine '%s' does not support the max-turns feature", engine.GetID())

After:

return fmt.Errorf("engine '%s' does not support 'max-turns'. Use an engine that supports this feature (e.g., copilot, claude)", engine.GetID())

Before (line 150):

return fmt.Errorf("max-continuations not supported: engine '%s' does not support the max-continuations feature", engine.GetID())

After:

return fmt.Errorf("engine '%s' does not support 'max-continuations'. Use an engine that supports this feature (e.g., copilot, claude)", engine.GetID())

Why This Matters

• User Impact: Removes documentation lookup; user can immediately switch to a supported engine
• Quality Factor: Actionable error messages with specific resolution guidance
• Frequency: Encountered when users experiment with engines or migrate workflows

Success Criteria

• Changes made to pkg/workflow/agent_validation.go only
• Error messages include at least one example of a supported engine
• Redundant phrasing removed
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/agent_validation.go
• No changes to other files required
• Can be completed independently

---

References: §23796357568

AI generated by Delight · history

• expires on Apr 7, 2026, 12:11 PM UTC

[github-actions[bot]]

💥 WHOOSH!

The Smoke Test Agent swoops in! ⚡

KABOOM! I, the mighty Claude engine, have swept through this repository like a bolt of lightning! Tests run! Build verified! Workflows checked!

ZAP! Every tool has been wielded with precision — Serena's LSP powers activated, Tavily's web-search beams fired, and the Playwright browser unleashed upon GitHub itself!

POW! Smoke test 23797519571 is IN PROGRESS. This agent was here! 🦸

— Your friendly neighborhood Smoke Test Agent 🕷️

💥 [THE END] — Illustrated by Smoke Claude · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #23880.