📚 Documentation Noob Test Report - March 24, 2026

[github-actions[bot]]

Summary

• Date: March 24, 2026
• Pages visited: Homepage, Quick Start, CLI Commands, Creating Workflows, Introduction (Overview, How They Work, Architecture), Examples, FAQ, Common Issues, Reference (Auth, Engines, Frontmatter)
• Overall impression: The docs have a solid structure with a clear Quick Start path, a detailed FAQ, and 74 navigation links with zero broken sidebar links. However, two bugs (grammar error, 404 slides page) and several "jargon ahead" UX gaps would slow down a first-time user.

---

🔴 Critical Issues Found

1. Grammar Error: Missing Verb in overview.mdx

File: docs/src/content/docs/introduction/overview.mdx — line 44

The sentence reads:

"The gh aw compile command this markdown file into a hardened GitHub Actions Workflow…"

The word "compiles" is missing. This is the first place a new user encounters an explanation of the core gh aw compile command. The broken sentence creates real confusion at a critical learning moment.

Fix:

- The `gh aw compile` command this markdown file into a hardened
+ The `gh aw compile` command **compiles** this markdown file into a hardened

2. Slides Page Returns HTTP 404

File: docs/src/content/docs/introduction/slides.mdx

The "Presentation Slides" page in the Introduction section contains a button linking to /gh-aw/slides/ which returns 404 Not Found. The file at /gh-aw/slides/index.html returns 200 — the directory index is just not served.

Fix: Add a redirect in astro.config.mjs:

  redirects: {
+   '/slides/': '/gh-aw/slides/index.html',

---

🟡 Confusing Areas

3. Quick Start Step 2: githubnext/agentics Not Explained

File: docs/src/content/docs/setup/quick-start.mdx

Step 2 immediately runs:

gh aw add-wizard githubnext/agentics/daily-repo-status
```

A brand new user has no idea what `githubnext/agentics` is. Is it a company? A package registry? A magic incantation? There's no link or explanation in the paragraph before this command.

**Questions a noob would ask:**
- _"What is githubnext/agentics?"_
- _"Is this downloading something from the internet onto my repo?"_
- _"Can I skip this and use my own workflow?"_

**Fix**: Add one sentence before the command:
> _"This fetches the pre-built Daily Repo Status workflow from the [`githubnext/agentics`](https://github.com/githubnext/agentics) community workflow library."_

#### 4. Absolute `github.github.com` URLs Instead of Relative Paths

**Files**: `quick-start.mdx`, `overview.mdx`, 15+ blog posts

The Quick Start "What's next" section links to:
```
https://github.github.com/gh-aw/blog/2026-01-12-welcome-to-pelis-agent-factory/
```

And `overview.mdx` uses:
```
https://github.github.com/gh-aw/reference/faq/#determinism

These absolute URLs work only when deployed to the production github.github.com domain. They silently break during local development (the page exists but the link opens an external URL that may not be reachable). Internal links should use relative paths.

Fix: Replace all https://github.github.com/gh-aw/ with /gh-aw/ across docs source files.

5. "Frontmatter" Used Without Inline Definition in Quick Start

File: docs/src/content/docs/setup/quick-start.mdx — Step 4

Step 4 says:

"If you have changed the frontmatter, regenerate the workflow YAML…"

"Frontmatter" is a Markdown/YAML term that many developers won't immediately recognize. The glossary link is available but easy to miss. A new user following the Quick Start would have to leave the page to understand this step.

Fix: Add inline clarification on first use:

"If you have changed the frontmatter (the YAML configuration section between --- markers at the top of the .md file), regenerate the workflow YAML…"

---

🟢 What Worked Well

• Clear 10-minute Quick Start with time estimate — the "⏱️ Estimated time: 10 minutes" banner sets good expectations and invites beginners to try it.
• Real workflow example on the homepage — the complete markdown snippet with frontmatter + instructions immediately shows what a workflow looks like. Excellent for first impressions.
• All 74 sidebar navigation links are valid — comprehensive audit found zero broken sidebar links.
• Comprehensive FAQ — answers many beginner questions clearly including: security, cost, determinism vs. non-determinism, cross-repo access, and max-turns vs. max-continuations.
• Good troubleshooting section — the Common Issues page covers installation failures, enterprise policy errors, and secrets configuration with specific error messages and step-by-step solutions.
• Multiple creation paths — web interface, CLI agent, and manual editing all documented side-by-side in Creating Workflows. Good for different user personas.

---

Recommendations

Quick Wins (minimal effort, high beginner impact)

1. Fix the grammar bug: Add "compiles" in overview.mdx line 44 — one word, big impact.
2. Fix slides redirect: Add one redirect in astro.config.mjs for /slides/.
3. Explain githubnext/agentics: Add one sentence in Quick Start Step 2.
4. Inline-define "frontmatter": Add parenthetical on first use in Quick Start Step 4.

Medium effort

5. Convert github.github.com absolute URLs to relative paths across docs and blog posts — grep -r "github.github.com/gh-aw" docs/src/ returns 15+ files.

Longer-term

6. Add a glossary tooltip integration — terms like "frontmatter", "lock file", "safe outputs", and "MCP" appear in beginner pages without context. An inline tooltip or <Aside> callout on first use would reduce "definition hunting".
7. Explain the compilation mental model earlier — the "you write markdown, the compiler produces YAML" concept is introduced in the overview but not reinforced in Quick Start. Adding a one-line diagram or callout in Step 4 would help.

---

Screenshots

---

References:

• §23491282388 — Workflow run that generated this report

AI generated by Documentation Noob Tester · history

• expires on Mar 25, 2026, 1:37 PM UTC

[github-actions[bot]]

🤖 Smoke test agent was here! 🚀\n\nBeep boop — the Copilot smoke test just rolled through and everything looks great. Nothing to see here, just a robot making sure the lights are on. ✅

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• Remove dual path safe output config generation #22687 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

📰 BREAKING: Report filed by Smoke Copilot · ◷

[github-actions[bot]]

💥 WHOOSH! 🦸 The Smoke Test Agent has arrived!

KAPOW! — Claude engine nominal. Run §23493201080 roared through all systems at full power!

ZAP! All MCP servers: ONLINE. Build: PASSED. Web: REACHABLE.

"With great agentic power comes great smoke-tested responsibility." 💥

— Claude Smoke Agent, 2026

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• Remove dual path safe output config generation #22687 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

💥 [THE END] — Illustrated by Smoke Claude · ◷

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-25T13:37:27.022Z.

Closed by Workflow