📚 Documentation Noob Test Report - December 5, 2025

[github-actions[bot]]

Summary

Date of test: December 5, 2025
Testing method: Direct markdown file analysis and curl-based HTML inspection (Playwright unavailable due to network restrictions)
Pages visited:

• Home page (/)
• Quick Start (/setup/quick-start/)
• CLI Commands (/setup/cli/)
• Creating Workflows (/setup/agentic-authoring/)
• Overview (/introduction/overview/)
• Examples - DailyOps (/examples/scheduled/dailyops/)
• Common Issues (/troubleshooting/common-issues/)
• Glossary (/reference/glossary/)

Overall impression: The documentation is comprehensive and well-structured, but suffers from "curse of knowledge" assumptions. The biggest blocker for new users is the hidden GitHub Copilot subscription requirement that only appears at Step 3. The documentation would greatly benefit from visual guidance (screenshots), clearer success criteria at each step, and upfront disclosure of all requirements.

---

🔴 Critical Issues Found (Block Getting Started)

(strong)1. Copilot Subscription Prerequisite Hidden(/strong) (HIGHEST PRIORITY)

Location: Quick Start, Step 3
Issue: The documentation mentions needing "Copilot Requests" permission for the PAT, but doesn't clearly state upfront that you MUST have an active GitHub Copilot subscription.

Impact: Users without Copilot invest 10-15 minutes following the guide, only to be blocked at Step 3 when they discover they need a paid subscription. This creates significant frustration and wasted time.

Fix:

• Add "Active GitHub Copilot subscription" to the "Must Have (before you start)" prerequisites section
• Include a direct link to https://github.com/settings/copilot to check subscription status
• Consider adding a note about free trial availability

Severity: 🔥 CRITICAL - Blocks all users without Copilot

(strong)2. Confusing "Agentic Setup" Section Placement(/strong)

Location: Quick Start, between Prerequisites and "Step 1"
Issue: The "Agentic Setup" section with the npx command appears BEFORE "Step 1 - Install the extension", making it unclear whether this is:

• A required step
• An alternative to the step-by-step guide
• An optional advanced feature

Impact: Users are confused about which path to follow. Some may try the npx command first and get confused, others may skip it wondering if they missed something important.

Fix:

• Move this section to the end as "Alternative: AI-Assisted Setup"
• OR clearly label it as "Optional: Quick Setup with Copilot CLI" if it appears before Step 1
• Add a sentence explaining when to use this vs. the manual steps

Severity: 🔴 HIGH - Creates confusion at the very start of the journey

(strong)3. Missing "What Repository Should I Use?" Guidance(/strong)

Location: Quick Start Prerequisites
Issue: The guide assumes users know which repository to work in, but provides no guidance on:

• Should I create a test repository?
• Can I use an existing repository?
• Does it need to be public or can it be private?
• Should it have any existing content?

Impact: Beginners may use the wrong repository, pollute a production repo with test workflows, or hesitate to start at all.

Fix: Add a clear "Choosing Your Repository" section:

## Choosing Your Repository

For this tutorial, we recommend creating a new **test repository**:
1. Go to https://github.com/new
2. Name it something like `gh-aw-test`
3. Make it public or private (both work)
4. Initialize with a README
5. Enable Discussions (Settings → Features → Discussions)

Severity: 🔴 HIGH - Causes hesitation and potential mistakes

(strong)4. Discussion Requirement Not Clear Upfront(/strong)

Location: Quick Start Prerequisites
Issue: Mentions "Discussions enabled" in a tip box but doesn't:

• Explain HOW to enable Discussions if not already on
• Show what it looks like when enabled
• Link to GitHub's official docs on enabling Discussions

Impact: Users reach Step 4, trigger the workflow, and it fails because Discussions aren't enabled. They must then backtrack to enable Discussions and re-run.

Fix:

• Add a direct link: How to enable Discussions
• Add a screenshot or clear steps: "Go to Settings → Features → Check 'Discussions'"
• Consider adding a warning: "⚠️ The sample workflow will fail if Discussions are not enabled"

Severity: 🔴 HIGH - Causes workflow failure and confusion

---

🟡 Confusing Areas (Slow Down Learning)

(strong)5. "Fine-Grained Token" Terminology Assumed(/strong)

Location: Quick Start, Step 3
Issue: Uses "fine-grained token" without explanation, assuming users know this vs. "classic token"

Impact: Users may create the wrong token type (classic PAT) and wonder why they don't see "Copilot Requests" permission.

Fix: Add a brief inline explanation or link:

Create a **fine-grained** [Personal Access Token](link to GitHub docs) 
(note: the newer "fine-grained" type, not "classic tokens")

(strong)6. Glossary Terms Overused in Quick Start(/strong)

Location: Quick Start, "Understanding Your First Workflow" section
Issue: Heavy use of linked glossary terms (frontmatter, YAML, compilation, permissions, safe-outputs) all at once

Impact: Cognitive overload - too many new concepts introduced simultaneously. Users may click away to read glossary entries and lose their place in the tutorial.

Fix:

• Reduce glossary links in the Quick Start
• Explain the most critical terms inline
• Save deep-dives for the "What's Next?" section
• Perhaps: "Don't worry about understanding all the configuration details yet - you'll learn more about frontmatter, permissions, and safe-outputs in the How It Works guide"

(strong)7. Compilation Step Timing Confusing(/strong)

Location: Quick Start, "Understanding Compilation" section
Issue: Explains compilation concept BEFORE adding the workflow, but compilation happens automatically when you use gh aw add --create-pull-request

Impact: Users wonder "when do I need to run gh aw compile?" since they never ran it manually in the tutorial.

Fix:

• Move the compilation explanation to AFTER Step 2
• Add a note: "Note: When you used gh aw add --create-pull-request, the compilation happened automatically. You only need to run gh aw compile manually when you edit the .md file yourself."

(strong)8. "Customize Your Workflow" Section Too Advanced(/strong)

Location: Quick Start, near the end
Issue: Jumps directly to npm installation, copilot CLI installation, and interactive agent sessions without explaining:

• What these tools do
• Why you'd want them
• Whether they're required for basic customization

Impact: Users copy-paste commands without understanding, or skip the section entirely thinking customization is too hard.

Fix: Simplify the basic path:

## Customize Your Workflow

The simplest way to customize your workflow:
1. Edit `.github/workflows/daily-team-status.md`
2. Change the natural language instructions
3. Run `gh aw compile` to update the .lock.yml file
4. Commit and push your changes

**Advanced**: You can also use GitHub Copilot CLI to help author workflows. [Learn more →](/setup/agentic-authoring/)

(strong)9. No Clear Success Indicators After "gh aw run"(/strong)

Location: Quick Start, Step 4
Issue: Says "check status with gh aw status" but doesn't show:

• What the expected output looks like
• How to interpret the status
• Where to find the created discussion on GitHub.com
• What "complete" vs "in progress" vs "failed" looks like

Impact: Users don't know if the workflow worked correctly or where to look for results.

Fix: Add expected output examples:

After a few moments, check the status:

```bash
gh aw status

You should see output like:

✓ daily-team-status    [enabled]  [completed successfully]
  Last run: 2 minutes ago
  Status: Success

Then, visit your repository's Discussions tab on GitHub.com to see the generated daily status report!

</details>

<details>
<summary>(strong)10. No Screenshots Throughout Quick Start(/strong)</summary>

**Location**: Entire Quick Start guide  
**Issue**: Zero screenshots showing what things should look like in the GitHub UI

**Impact**: Users can't verify they're in the right place or that things are working correctly. Visual learners especially struggle.

**Fix**: Add 3-5 key screenshots:
1. Enabling Discussions in repository settings
2. Creating a fine-grained PAT with "Copilot Requests" highlighted
3. Adding the secret to repository settings
4. A successful workflow run in GitHub Actions
5. The created discussion in the Discussions tab

**Priority**: Medium-High (would significantly improve UX)
</details>

<details>
<summary>(strong)11. "Network: defaults" Not Explained(/strong)</summary>

**Location**: Quick Start, workflow example  
**Issue**: Shows `network: defaults` in the frontmatter example without any explanation

**Impact**: Users see configuration they don't understand and may worry about security implications.

**Fix**: Either:
- Remove it from the initial example (if not essential)
- Add a brief inline comment: `network: defaults  # Allows AI to access GitHub API`
- Link to network documentation for details
</details>

<details>
<summary>(strong)12. Public vs Private Repository Confusion(/strong)</summary>

**Location**: Quick Start, Step 3 (PAT creation)  
**Issue**: Instructs users to select "Public repositories" for the PAT, but doesn't clarify:
- Does this mean MY repository must be public?
- Or is this just a token scope setting?

**Impact**: Users with private repositories may think the tool won't work for them.

**Fix**: Clarify the language:
```markdown
Under "Repository access," select **"Public repositories"** (this is a token scope 
requirement - your actual repository can be either public or private)

(strong)13. Missing Error Handling Guidance(/strong)

Location: Throughout Quick Start
Issue: No mention of what to do when steps fail or produce errors

Impact: Users get stuck when something goes wrong and don't know where to look for help.

Fix: Add troubleshooting callout boxes at key steps:

:::tip[Troubleshooting]
If `gh extension install` fails, try the standalone installer:
```bash
curl -sL https://raw.githubusercontent.com/githubnext/gh-aw/main/install-gh-aw.sh | bash

See Common Issues for more help.
:::

</details>

<details>
<summary>(strong)14. Examples Lack "Try This" Instructions(/strong)</summary>

**Location**: Example pages (DailyOps, IssueOps, etc.)  
**Issue**: Examples explain patterns but don't provide easy copy-paste commands to try them

**Impact**: Users must figure out how to install and test examples themselves.

**Fix**: Add "Try This Example" boxes:
```markdown
## Try This Example

Install this workflow in your repository:
```bash
gh aw add githubnext/agentics/daily-team-status --create-pull-request

Then customize it to fit your needs.

</details>

<details>
<summary>(strong)15. CLI Commands Page Overwhelming Length(/strong)</summary>

**Location**: /setup/cli/  
**Issue**: Despite having a helpful "Most Common Commands" section at the top, the full page is 440+ lines long

**Impact**: Intimidating for beginners browsing the documentation.

**Fix**: Consider splitting into:
- **Essential Commands** page (init, add, compile, run, status)
- **Advanced Commands** page (trial, mcp, pr, etc.)
- Or use a tabbed interface to hide advanced commands by default
</details>

---

## 🟢 What Worked Well

The documentation has many strengths worth highlighting:

1. **Clear Value Proposition**: Home page effectively communicates what the tool does and why you'd want it
2. **"Getting Started" Button**: Prominent, easy to find on the home page
3. **Prerequisites Organization**: Good use of nested callouts for "Must Have," "Must Configure," and "Will Need Later"
4. **Research Prototype Badge**: Clear warning that this is experimental - sets expectations appropriately
5. **Compilation Diagram**: The `.md → compile → .lock.yml` visualization is helpful
6. **"Most Common Commands"**: CLI page highlights the 5 essential commands - great for 95% of users
7. **Glossary Quality**: Well-written definitions with good examples
8. **Code Examples**: Plenty of bash and YAML snippets throughout
9. **Alternative Installation**: Standalone installer option for Codespaces users
10. **Security Emphasis**: Good attention to security considerations throughout

---

## Recommendations

### High Priority (Fix These First)

1. **🔥 Move Copilot Subscription to Prerequisites** - Make it crystal clear this requires an active Copilot subscription BEFORE users start
   
2. **Add "Choose Your Repository" Section** - Guide users on creating a test repo or choosing an existing one
   
3. **Clarify Agentic Setup Section** - Move after main steps or clearly label as optional/alternative
   
4. **Add Output Examples** - Show what success looks like at each step with example command outputs
   
5. **Include 3-5 Key Screenshots** - Token creation, workflow run status, created discussion

### Medium Priority (Quick Wins)

6. **Add Troubleshooting Callouts** - Inline help for common failures at each step
   
7. **Simplify Initial Workflow Example** - Remove advanced features from first example, add them later
   
8. **Clarify Token Scope Requirements** - Explain the public/private repository confusion
   
9. **Link to Enabling Discussions** - Add direct link with clear instructions
   
10. **Reduce Glossary Links in Quick Start** - Explain critical terms inline instead

### Lower Priority (Nice to Have)

11. **Add Video or GIF Demo** - Show workflow in action on home page (5-minute demo)
   
12. **Split CLI Commands Page** - Separate essential from advanced commands
   
13. **Enhance Examples with "Try This"** - Add installation commands to example pages
   
14. **Categorize Glossary** - Split into "Beginner" and "Advanced" sections
   
15. **Add Success Checklist** - "You'll know it worked when you see..."

---

## Simulated Beginner Journeys

### Journey 1: First-Time User (No Copilot)
1. ✅ Finds "Getting Started" button easily
2. ✅ Understands it's a research prototype  
3. ⚠️ Confused by "Agentic Setup" appearing before Step 1
4. ✅ Installs gh CLI and extension successfully
5. ⚠️ Slightly confused by compilation explanation timing
6. ✅ Adds workflow with `gh aw add`
7. ❌ **BLOCKED** - Reaches Step 3, discovers they need Copilot subscription
8. 🔄 Must sign up for Copilot (paid), return later

**Time invested before blocking**: ~15 minutes  
**Frustration level**: High (should have been told upfront)

### Journey 2: Experienced Developer (Has Copilot)
1. ✅ Breezes through prerequisites
2. ⚠️ Slightly confused by PAT creation UI nuances
3. ✅ Successfully adds and configures workflow
4. ⚠️ Uncertain if workflow ran correctly - output not clear
5. ⚠️ Has to hunt for the created discussion
6. ✅ Eventually figures it out

**Time to first success**: ~20-25 minutes  
**Frustration level**: Medium (got there but with some confusion)

### Journey 3: Non-Native English Speaker
1. ⚠️ Lots of jargon: "agentic," "frontmatter," "MCP," "compilation"
2. ⚠️ Would greatly benefit from more visual guidance
3. ⚠️ Technical terms used before being explained
4. ✅ Code examples are helpful (universal language)

**Experience**: Would benefit most from screenshots and simpler language

---

## Conclusion

The GitHub Agentic Workflows documentation is **comprehensive and well-intentioned**, but suffers from the classic "curse of knowledge" problem - the authors know the product so well that they make assumptions about what users know.

**Key Insight**: The biggest blocker isn't complexity - it's the hidden Copilot subscription requirement. Users invest significant time before discovering this prerequisite, creating unnecessary frustration.

**Estimated Time to First Success**:
- **Current state**: 30-45 minutes (if you already have Copilot)
- **With recommended improvements**: 10-15 minutes

**Biggest Blocker**: Reaching Step 3 and discovering you need a paid Copilot subscription

**Biggest Confusion**: Not knowing what success looks like after running the workflow

**Easiest Win**: Add Copilot subscription to prerequisites and include 3-5 screenshots showing the UI at key steps

---

## Test Limitations

**Note**: This analysis was conducted using direct file inspection and curl-based HTML analysis rather than interactive Playwright browser automation (which was unavailable due to network restrictions). A follow-up test with visual browser automation would provide additional insights into:
- Navigation flow and UX issues
- Visual hierarchy and layout problems  
- Interactive element discoverability
- Mobile responsiveness
- Accessibility issues

Despite this limitation, the text-based analysis reveals significant content and structure issues that should be addressed regardless of visual presentation.


> AI generated by [Documentation Noob Tester](https://github.com/githubnext/gh-aw/actions/runs/19959535272)

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 3 days ago.