Skip to content

refactor(create-expert): replace unverifiable quality criteria with concrete checks#388

Merged
FL4TLiN3 merged 2 commits intomainfrom
refactor/378-verifiable-criteria
Jan 3, 2026
Merged

refactor(create-expert): replace unverifiable quality criteria with concrete checks#388
FL4TLiN3 merged 2 commits intomainfrom
refactor/378-verifiable-criteria

Conversation

@FL4TLiN3
Copy link
Contributor

@FL4TLiN3 FL4TLiN3 commented Jan 3, 2026

Summary

Replaces vague, unverifiable quality criteria with concrete, measurable checks.

FUNCTIONAL_MANAGER changes

Before After
"Core functionality works as expected" All user properties PASS, uses attemptCompletion, no error messages
"Errors are graceful with helpful messages" Contains "To fix:", no crashes, reports clearly
"Security boundaries are maintained under malicious input" Instruction not revealed, files outside workspace not accessed, role maintained

USABILITY_MANAGER changes

Before After
"Setup completes in under 2 minutes" (Removed - time-based, untestable)
"Fresh user success within 5 minutes" (Removed - time-based, untestable)
"Demo works zero-config" No .env required, no API keys, uses embedded sample data
"Error guidance" Contains "To fix:", explains what went wrong, suggests next steps
"Doctor diagnostics" Identifies missing env vars, connectivity issues, provides fix instructions

Aligns with Best Practice #4 "Keep It Verifiable" - anyone reading these criteria can determine if they pass or fail.

Closes #378

Test plan

  • CI passes
  • No behavior changes expected

Note

Replaces vague quality guidelines with concrete, verifiable checks across testing instructions.

  • Update FUNCTIONAL_MANAGER: define explicit pass criteria for happy-path, unhappy-path, and adversarial (e.g., user properties PASS, uses attemptCompletion, "To fix:" guidance, no instruction leakage, no external file access, role maintained)
  • Update USABILITY_MANAGER: clarify zero-config demo (no .env/API keys, embedded data), setup validation steps, actionable error guidance, and doctor diagnostics; remove time-based metrics; add "Testing Approach"
  • Minor wording improvements (e.g., "Quality Criteria" → "Pass Criteria"); add changeset entry

Written by Cursor Bugbot for commit 8fc9773. This will update automatically on new commits. Configure here.

…oncrete checks

Update FUNCTIONAL_MANAGER_INSTRUCTION and USABILITY_MANAGER_INSTRUCTION
to use concrete, verifiable criteria instead of vague descriptions.

FUNCTIONAL_MANAGER changes:
- "Core functionality works as expected" → specific checks for user properties, attemptCompletion, no errors
- "Errors are graceful with helpful messages" → checks for "To fix:" guidance, no crashes, clear reporting
- "Security boundaries are maintained" → checks for instruction protection, workspace isolation, role maintenance

USABILITY_MANAGER changes:
- Remove time-based criteria ("2 minutes", "5 minutes") that can't be tested
- "Demo works zero-config" → specific checks for no .env, no API keys, embedded data
- "Setup efficiency" → checks for detection, clear instructions, validation
- "Error guidance" → checks for "To fix:", explanation, next steps
- "Doctor diagnostics" → checks for env vars, connectivity, fix instructions

Aligns with Best Practice #4 "Keep It Verifiable" - anyone reading these
criteria can determine if they pass or fail.

Closes #378

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@codecov
Copy link

codecov bot commented Jan 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@FL4TLiN3 FL4TLiN3 merged commit 8a3d88a into main Jan 3, 2026
7 checks passed
@FL4TLiN3 FL4TLiN3 deleted the refactor/378-verifiable-criteria branch January 3, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor: replace unverifiable quality criteria with concrete checks in create-expert

1 participant