Skip to content

v2: Skill doc rewriter - generate improved docs from model feedback #1

@EmZod

Description

@EmZod

Feature

After running a focus group test, generate an improved version of the skill document that incorporates model feedback.

Current Behavior

Models report:

  • Confusions
  • Potential failures
  • Suggested improvements

User manually updates their skill doc based on this feedback.

Proposed Behavior

After collecting model feedback, synthesize it into a rewritten skill document that:

  1. Fixes identified confusions - Clarify ambiguous sections that multiple models flagged
  2. Optimizes progressive disclosure - Restructure so agents get essential info first, details later
  3. Improves examples - Add/improve examples where models indicated confusion
  4. Better references - Add cross-references and context where models needed more info
  5. Clearer structure - Reorganize based on how models actually parse the document

Output

focus-group test ./SKILL.md "task" --rewrite

# Outputs:
# 1. Normal model feedback (current behavior)
# 2. NEW: ./SKILL.improved.md - rewritten skill doc

Implementation Notes

  • Use a capable model (e.g., claude-3-5-sonnet) for synthesis
  • Feed it: original doc + all model feedback + rewrite instructions
  • May need structured output from test models to make synthesis reliable
  • Consider iterative refinement (run focus group on improved doc)

Success Criteria

  • Improved doc scores better on subsequent focus group runs
  • Reduced "confusions" count across models
  • Maintains original intent while improving clarity

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions