📚 Documentation Noob Test Report - November 24, 2025

[github-actions[bot]]

📚 Documentation Noob Test Report

Date: November 24, 2025
Tester Persona: Complete beginner with no prior experience with GitHub Agentic Workflows
Pages Analyzed: 7 key documentation pages
Testing Method: Manual review of source documentation files as a first-time user

---

Executive Summary

As a complete beginner trying to get started with GitHub Agentic Workflows, I navigated through the documentation to evaluate the onboarding experience. The documentation shows strong fundamentals with excellent Quick Start content, but has several areas that could confuse newcomers, particularly around jargon and conceptual understanding.

Overall Rating: ⭐⭐⭐⭐☆ (4/5 - Good, but room for improvement)

---

🔴 Critical Issues (Blocking Getting Started)

1. "Frontmatter" Not Defined Before First Use

Location: Quick Start Guide (Step 2, line 47)

Issue: The term "frontmatter" appears in the middle of instructions without prior definition. It's mentioned in line 47, but only explained much later in line 136.

Impact: Beginners see a "frontmatter" link before understanding what it is, creating unnecessary confusion at a critical moment.

Fix: Define frontmatter BEFORE showing the workflow example, or add a callout box:

:::note[What is frontmatter?]
The configuration section at the top of a workflow file, between `---` markers, containing settings like triggers and permissions.
:::

2. Unclear Token Permissions Requirements

Location: Quick Start Guide, Step 3 (lines 66-68)

Issue:

• Step says "select your user account (not an organization, see note below)" but there's no note explaining WHY
• Users will wonder: "Can I use organization tokens later? What's the limitation?"

Impact: Causes confusion and uncertainty about token setup. Users may waste time trying to figure out if they did something wrong.

Fix: Add a clear note explaining the limitation:

:::caution[Why user account only?]
Currently, fine-grained PATs with Copilot Requests permission can only be created under user accounts, not organizations. For organization workflows, each team member needs their own token.
:::

3. Missing Visual Flow/Architecture Diagram

Location: Home page and "How It Works"

Issue: No visual diagram showing the overall architecture or flow:
Markdown → Compilation → YAML → GitHub Actions → AI Agent

Impact: Beginners struggle to build a mental model of how the system works. Text-only explanations make it harder to grasp the big picture.

Fix: Add a simple flowchart on "How It Works" page showing the complete journey from writing a workflow to execution.

---

🟡 Confusing Areas (Slow Down Learning)

4. "Agentic Workflow" - Circular Definition

Location: Multiple pages

Issue: The glossary defines "agentic workflow" as "AI-powered automation that can make decisions" but this doesn't clearly explain WHAT makes it "agentic" versus regular automation.

Impact: Beginners don't grasp the core differentiator of this technology.

Suggestion: Improve the definition with a comparison:

**Agentic Workflow:** An AI-powered automation where the AI makes intelligent decisions based on context, rather than following a rigid script. Like having a smart assistant who understands your intent and adapts their actions, versus a simple script that does the same thing every time.

5. Compilation Process Needs Better Explanation

Location: Quick Start, Step 2

Issue: While there's a "Why Compile?" section, it doesn't explain:

• Do I compile locally or does GitHub do it?
• When do I need to recompile?
• What if compilation fails?

Impact: Users understand the WHAT but not the HOW or WHEN.

Better version:

### Why Compile?

GitHub Actions can only run YAML workflows. Your human-friendly `.md` file needs to be converted to `.lock.yml` (YAML format) that Actions can execute.

**When to compile:**
- After creating a new workflow
- After editing an existing workflow
- Before committing changes

**How to compile:**
```bash
gh aw compile

Where does it happen?
Compilation happens locally on your machine using the gh aw CLI. The compiled .lock.yml file is committed to your repository alongside the .md file.

### 6. Safe Outputs Concept Not Clear

**Location:** Quick Start and Overview  

**Issue:** "safe-outputs" is mentioned but not well explained:
- WHY do we need safe outputs?
- What's the security benefit?
- What happens if I don't use safe-outputs?

**Impact:** Users copy-paste configuration without understanding critical security implications.

**Fix:** Add explanation in Quick Start:
```markdown
:::tip[Why Safe Outputs?]
Safe outputs let the AI create issues, PRs, and discussions WITHOUT giving it direct write access to your repository. This prevents malicious or buggy AI behavior from modifying your code directly. The AI proposes actions, and GitHub safely executes them after validation.
:::

7. "Custom Agent" vs "AI Agent" Confusion

Location: Creating Workflows page

Issue: The page mentions three similar-sounding but different concepts:

• "custom agent" (specialized AI instructions)
• "AI agent" (the thing executing workflows)
• "coding agent" (GitHub Copilot CLI)

A beginner can't tell these apart.

Fix: Add glossary at top of page:

**Key Terms:**
- **AI Agent**: The AI (Claude, Codex, etc.) that executes your workflow
- **Custom Agent**: A specialized instruction file that customizes how AI behaves (like a template)
- **Coding Agent**: The specific AI tool we use (GitHub Copilot CLI by default)

8. Copilot CLI vs GitHub Copilot Chat Confusion

Location: Quick Start Step 4 and Creating Workflows

Issue: The documentation mentions multiple tools without clarifying their relationship:

• gh aw commands
• "Copilot CLI" with npm install -g @github/copilot-cli``
• "VS Code with GitHub Copilot Chat"

Impact: Beginners don't understand which tool does what and how they relate to each other.

Fix: Add a "Tools Overview" section early in Quick Start explaining the ecosystem.

9. No Clear "Next Steps" Path

Location: End of Quick Start

Current text:

"Use [Authoring Agentic Workflows] to create workflows with AI assistance, explore more samples in the [agentics] repository..."

Issue: Too many options presented at once with no clear recommended path for different learning styles.

Better version:

## What's Next?

Choose your learning path:

1. **For hands-on learners**: Try modifying the daily team status workflow you just added
   - Edit `.github/workflows/daily-team-status.md`
   - Change the schedule or customize the report content
   - Run `gh aw compile` and test your changes

2. **For example-driven learners**: Browse the [Examples gallery](/gh-aw/examples/) to see workflows in action

3. **For deep-dive learners**: Read [How It Works](/gh-aw/introduction/how-it-works/) to understand the architecture

4. **For creators**: Use [Creating Workflows](/gh-aw/setup/agentic-authoring/) to build your own from scratch

---

🟢 What Worked Well

✅ Excellent Prerequisites Section

The Quick Start has a fantastic checklist of prerequisites. Clear, actionable, with verification steps ("Run gh --version"). This is exactly what beginners need!

✅ Good Warning Banners

The RESEARCH PROTOTYPE badge and warning callouts set proper expectations. Users know this is experimental.

✅ Code Examples Are Abundant

Every page has relevant code examples. The Quick Start shows complete, working commands that can be copy-pasted.

✅ Good Progressive Disclosure

The documentation doesn't dump everything at once. Glossary links let users dive deeper when ready.

✅ Clear Installation Steps

Step-by-step installation with actual commands is perfect. No ambiguity about what to run.

✅ Helpful Inline Definitions

Many concepts have inline parenthetical explanations like:

"using [agentic workflows] (AI-powered automation that can make decisions)"

This is great! Just needs to be consistent everywhere.

✅ Realistic Examples

The "daily team status" example is practical and relatable. Not a contrived "hello world" - it provides actual business value.

---

📊 Navigation and Structure Analysis

Home Page

• ✅ Clean, professional, clear "Getting Started" CTA
• ⚠️ "Continuous AI" in features - what does this mean?
• ❌ Missing a one-sentence "what is this?" above the fold

Quick Start

• ✅ Length (~1500 words) is appropriate for getting started
• ✅ Logical step-by-step progression
• ✅ 8 code examples - excellent
• ❌ Missing troubleshooting section for common failures

Creating Workflows

• ⚠️ Too many concepts at once (custom agents, VS Code, Copilot CLI, dictation)
• 💡 Better structure: Start with ONE simple method, then show alternatives

---

🎯 Recommendations

Priority 1: Quick Wins (Do These First) 🚀

1. Add inline definitions for all jargon on first use

   • Frontmatter, safe-outputs, compilation, MCP
   • Use this pattern: term (simple explanation in plain English)

2. Add troubleshooting section to Quick Start

   ## Troubleshooting
   
   **Error: "Failed to compile workflow"**
   - Check that your frontmatter has valid YAML syntax
   - Ensure all required fields are present (on, permissions)
   
   **Error: "No such workflow"**
   - Verify you merged the PR from Step 2
   - Check that the workflow file exists in `.github/workflows/`
   
   **Workflow runs but produces no output**
   - Check Actions logs: `gh aw status`
   - Verify your COPILOT_GITHUB_TOKEN is set correctly

3. Add a "Tools Overview" diagram showing how these relate:

   • gh CLI
   • gh-aw extension
   • GitHub Copilot CLI
   • VS Code + GitHub Copilot Chat

4. Improve "What's Next?" section with clear paths for different learning styles

5. Add visual architecture diagram to "How It Works" page

Priority 2: Important Improvements 📈

6. Create a "Common Terms" page or improve glossary with beginner-friendly definitions and analogies

7. Add "Expected Time" estimates to Quick Start steps

   ### Step 1 — Install the extension ⏱️ 2 minutes

8. Add success indicators after each step

   ✅ **Success!** You should see: "Extension githubnext/gh-aw installed"

9. Video walkthrough - A 5-minute video showing the Quick Start process would be invaluable for visual learners

10. Add FAQ section addressing common questions:

    • Cost (does this use my Copilot subscription?)
    • Privacy (what data does the AI see?)
    • Quotas/limits
    • Production readiness

Priority 3: Nice to Have 💎

11. Interactive tutorial - Embedded code playground for trying workflows

12. Migration guide - "If you know GitHub Actions, here's what's different"

13. Comparison table - Agentic Workflows vs traditional Actions vs GitHub Apps

14. Community examples - User-submitted workflows gallery

---

🎓 Beginner Testing Results

I asked myself 5 key questions a beginner would have:

Question	Answer in Current Docs	Suggested Fix
What is this in one sentence?	Not immediately clear from home page	Add hero subtitle: "Write AI-powered GitHub automations in plain English instead of complex YAML"
How much does this cost?	Not mentioned anywhere	Add to prerequisites or FAQ
Can I use this in production?	Warning says "research demonstrator" but doesn't clearly say YES or NO	Add explicit guidance on production use
What's the difference between this and GitHub Actions?	Explained in Overview, but not in Quick Start	Add one-liner comparison in Quick Start
Do I need to know Python/JavaScript/YAML?	Not addressed	Add to prerequisites or intro

---

📋 Getting Started Success Checklist

Testing my ability to complete the Quick Start as a beginner:

• ✅ Found the Quick Start guide easily
• ✅ Understood prerequisites
• ✅ Successfully followed installation commands (conceptually)
• ⚠️ Partially understood what "compilation" means
• ⚠️ Partially understood security model (safe-outputs)
• ❌ Confused about different AI tools (Copilot CLI vs Chat vs "agent")
• ✅ Understood basic workflow structure
• ⚠️ Uncertain about next steps after Quick Start

Success Rate: 5/8 = 62.5%

---

🎯 Final Verdict

The GitHub Agentic Workflows documentation is well-structured and comprehensive with an excellent Quick Start guide that successfully gets users to their first workflow. However, it makes several assumptions about user knowledge that could slow down or confuse complete beginners.

Key Strengths:

• ✅ Clear, sequential installation steps
• ✅ Abundant, practical code examples
• ✅ Real-world example workflow with business value
• ✅ Appropriate use of warnings and expectations-setting

Areas for Improvement:

• ❌ Define technical jargon before using it
• ❌ Add visual aids (diagrams, architecture overview)
• ❌ Clarify tool relationships and ecosystem
• ❌ Better explain security model and benefits
• ❌ Add troubleshooting guidance
• ❌ Provide clearer next steps and learning paths

Recommended Action: Implement Priority 1 quick wins immediately. These are small documentation changes that will dramatically improve the beginner experience without requiring major restructuring.

---

📝 Pages Analyzed

1. Home Page (index.mdx)
2. Quick Start Guide (setup/quick-start.md)
3. CLI Commands (setup/cli.md)
4. Creating Workflows (setup/agentic-authoring.mdx)
5. Overview (introduction/overview.mdx)
6. How It Works (introduction/how-it-works/)
7. Example: IssueOps (examples/issue-pr-events/issueops/)

---

Report Generated: November 24, 2025
Workflow: Documentation Noob Testing
Testing Approach: First-time user perspective, manual review of documentation source files
Status: Complete ✅

Labels: documentation, user-experience, automated-testing

AI generated by Documentation Noob Tester

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.