📚 Documentation Noob Test Report - November 23, 2025

[github-actions[bot]]

📚 Documentation Noob Test Report - November 23, 2025

Executive Summary

As a complete beginner to GitHub Agentic Workflows, I navigated the documentation site to identify pain points in the getting started experience. This report documents critical issues, confusing areas, and recommendations for improving the new user experience.

Date of Test: November 23, 2025
Pages Visited:

• Homepage: /gh-aw/
• Quick Start: /gh-aw/setup/quick-start/
• CLI Commands: /gh-aw/setup/cli/
• Creating Workflows (Agentic Authoring): /gh-aw/setup/agentic-authoring/
• Example (Coding & Development): /gh-aw/examples/issue-pr-events/coding-development/
• Frontmatter Reference: /gh-aw/reference/frontmatter/

Overall Impression:
The documentation is comprehensive but has several barriers for beginners. The biggest issue is the confusing onboarding flow - there are multiple entry points ("Quick Start" vs "Creating Workflows") that teach different initial steps, and key concepts like "frontmatter" and "compilation" are used before being explained.

---

🔴 Critical Issues (Block Getting Started)

1. Confusing Multiple Entry Points for Getting Started

Issue: The documentation has two different "getting started" paths that teach conflicting first steps:

• Quick Start (/setup/quick-start/) says: "Install extension → Add sample workflow → Add AI secret → Run"
• Creating Workflows (/setup/agentic-authoring/) says: "Run gh aw init first"

Why This Blocks Users:

• A beginner clicking "Getting Started" on the homepage goes to Quick Start
• Quick Start never mentions gh aw init until Step 4 (under "Customize Your Workflow")
• But Agentic Authoring says gh aw init is the "Quick Start" and should be done first
• This creates confusion: "Did I skip a critical step? Should I go back?"

Specific Example:
Quick Start says:

gh extension install githubnext/gh-aw
gh aw add githubnext/agentics/daily-team-status --pr

But Creating Workflows says:

gh aw init  # Do this FIRST

Recommendation:

• Either merge these into ONE getting started guide with clear sequential steps
• OR clearly label them as "Quick Start (Try it in 5 minutes)" vs "Full Setup (Recommended for teams)"
• The Quick Start should mention: "Note: This is a quick demo. For production use, start with gh aw init"

2. "Personal Access Token (PAT)" - Term Used Before Explained

Issue: Step 3 of Quick Start says:

"create a fine-grained Personal Access Token (PAT)"

Why This Blocks Users:

• The term "PAT" appears in parentheses as if it's assumed knowledge
• A beginner doesn't know what a PAT is or why they need it
• The definition is buried in a later sentence: "(a secure key for API access)"
• But this definition appears BEFORE the actual explanation, in the Prerequisites section

Recommendation:
In Step 3, add a beginner-friendly explanation:

### Step 3 — Add an AI secret

To run AI workflows, you need to give GitHub Actions access to an AI service. 
We'll create a **Personal Access Token (PAT)** - think of it as a password 
that lets GitHub Actions use GitHub Copilot on your behalf.

3. "Compilation" Concept Not Explained Before Use

Issue: Quick Start Step 2 says:

"This creates a pull request that adds .github/workflows/daily-team-status.md and the compiled .lock.yml"

Why This Blocks Users:

• A beginner sees "compiled" but doesn't know what compilation means in this context
• The term "compilation" is linked to the glossary, but beginners don't know to click it
• The explanation "Why Compile?" comes AFTER this mention

Recommendation:
Move the "Why Compile?" section BEFORE Step 2, explaining the concept before it's used.

4. Unclear What "Frontmatter" Means

Issue: Throughout the docs, "frontmatter" is used as if everyone knows what it means. The term appears 15+ times before being defined.

Why This Blocks Users:

• Complete beginners (especially non-Jekyll/non-Astro users) don't know "frontmatter"
• It's web development jargon that's not universal

Recommendation:
At the FIRST use of "frontmatter" in Quick Start, add:

The section between the `---` markers is called **frontmatter** (the YAML 
configuration at the top of the file, similar to metadata in a document).

---

🟡 Confusing Areas (Slow Down Learning)

5. Too Many Concepts at Once in Step 3

Issue: Step 3 (Add an AI secret) introduces:

• Personal Access Tokens
• Fine-grained tokens vs classic tokens (implied)
• Repository secrets
• GitHub Actions secrets
• The gh secret set command
• The -a actions flag
• The concept of "Resource owner" (user vs organization)

Why This Slows Learning:

• This is the HARDEST step for beginners
• Each bullet point introduces a new concept
• The note about "Resource owner" adds confusion without explanation

Recommendation:

• Simplify Step 3 to bare minimum for first-time users
• Move advanced topics (org tokens, alternative engines) to "What's Next" or a dedicated page

6. Glossary Terms Need Inline Definitions

Issue: Throughout Quick Start, terms are linked to the glossary but not defined inline.

Why This Slows Learning:

• Users shouldn't have to click away to understand the current page
• It breaks the reading flow
• Mobile users especially hate this pattern

Recommendation:
Use inline definitions with glossary links:

The [**agentic workflow**](/gh-aw/reference/glossary/#agentic-workflow) 
(AI-powered automation that can make decisions) requires careful attention...

7. "Coding Agent" Term Appears Without Context

Issue: Step 3 says:

"Agentic workflows use a coding agent (the AI that executes your workflow instructions): GitHub Copilot CLI (default)."

Recommendation:
Flip the order to define before using:

Agentic workflows need AI to execute the instructions you write. We call this the 
**coding agent**. By default, we use GitHub Copilot CLI.

8. The .lock.yml Filename Convention is Unclear

Issue: The docs mention .lock.yml but don't show the full path clearly.

Recommendation:
Always show the full path pair:

This creates a pull request that adds two files:
- `.github/workflows/daily-team-status.md` (your editable workflow)
- `.github/workflows/daily-team-status.lock.yml` (auto-generated GitHub Actions file)

9. Code Examples Use Flags Without Explanation

Issue: Commands like gh aw add githubnext/agentics/daily-team-status --pr don't explain what --pr means.

Recommendation:
Add inline comments:

# Add the workflow and create a PR (instead of committing directly)
gh aw add githubnext/agentics/daily-team-status --pr

---

🟢 What Worked Well

✅ Clear Prerequisites Checklist

The checklist format in Prerequisites is excellent - helps users verify they're ready before starting.

✅ "Why Compile?" Section

Great explanation of a complex concept:

"Think of it like: Markdown (what you write) → YAML (what Actions runs)"

This pattern should be used more throughout the docs.

✅ Progressive Disclosure in CLI Commands

The "🚀 Most Common Commands" section at the top is fantastic. "95% of users only need these 5 commands" helps beginners focus on what's important.

✅ Warning Banner

The warning about "research demonstrator" sets proper expectations.

✅ Code Blocks Have Good Formatting

All code blocks are properly formatted with syntax highlighting and copy buttons.

---

Recommendations

🎯 Quick Wins (Could Help Immediately)

1. Merge or Clearly Separate Quick Start vs Creating Workflows

   • Option A: Merge into one "Getting Started Guide" with gh aw init as Step 1
   • Option B: Rename to "5-Minute Demo" vs "Production Setup"

2. Add Inline Definitions for Jargon (15 minutes of work)

   • Find all glossary-linked terms in Quick Start
   • Add inline definitions: "term (short definition)"

3. Reorder "Why Compile?" to Come First (5 minutes)

   • Move it before Step 2
   • This explains a concept BEFORE using it

4. Add Comments to Code Examples (10 minutes)

   • Explain what each flag does
   • Help users understand, not just copy-paste

5. Fix the "Resource Owner" Note (2 minutes)

   • Either add the missing note explaining why "not an organization"
   • Or remove "see note below" if there's no note

📈 Medium-Term Improvements

6. Create a "Concepts" Page

   • Explain: compilation, frontmatter, coding agents, safe-outputs
   • Link from Quick Start: "New to these concepts? Read [Understanding Agentic Workflows] first"

7. Add Troubleshooting to Each Step

   • "What if gh extension install fails?"
   • "What if I don't see 'Copilot Requests' permission?"

8. Add Screenshots for Complex Steps

   • Step 3 (creating PAT) would benefit from screenshots
   • Show what "Copilot Requests" permission looks like

9. Create a "Common Mistakes" Section

   • "Forgot to run gh aw compile after editing .md file"
   • "Used organization token instead of personal token"

10. Add a Flowchart on Homepage

    Write Markdown → Compile → GitHub Actions → AI Executes → Results
    

🚀 Longer-Term Improvements

11. Create a Video Walkthrough (5-minute video showing Quick Start end-to-end)
12. Add "What Happens Next?" to Each Step (helps users verify completion)
13. Create a "From Zero to Hero" Tutorial (multi-day learning path)
14. Add Interactive Playground (test markdown → YAML compilation in browser)
15. Improve Search/Discoverability (tag content by skill level)

---

Conclusion

The documentation is comprehensive and well-structured, but it suffers from a classic documentation problem: it was written by experts for experts.

The biggest barrier for new users is the lack of a single, clear, sequential path from "I've never heard of this" to "I ran my first workflow successfully."

Priority Fixes:

1. Create ONE definitive getting started guide with concepts explained before use
2. Add inline definitions for ALL jargon in the getting started path
3. Add visual aids (screenshots, diagrams) for confusing steps

With these changes, new users would have a much smoother onboarding experience.

---

Test Methodology

This test was conducted by:

1. Building the documentation site locally
2. Analyzing the rendered HTML structure
3. Reading through the source markdown files
4. Applying a "beginner's mindset" to identify confusion points
5. Documenting specific examples of unclear or blocking content

---

Labels: documentation, user-experience, automated-testing

AI generated by Documentation Noob Tester

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.