skill-eval

Evaluation framework for AI agent skills — measure whether capabilities actually work before they ship.

---

Why

ClawHub has 13,729 skills. 67% of them fail in practice. The ecosystem needs a way to answer: does this skill actually do what it claims?

skill-eval is a framework for writing, running, and scoring evaluations against agent skills. It draws on:

• The ClawHub corpus analysis — empirical failure patterns across 13K+ skills
• Anthropic's skill evaluation patterns in Claude Code
• mgechev/skill-eval — open-source skill assessment methodology
• The effector-spec type system for capability interfaces

What It Measures

Structural Quality (static, no execution)

Metric	What it checks	Weight
frontmatter_completeness	All required YAML fields present and valid	0.15
section_coverage	Purpose, When to Use, When NOT to Use, Setup, Commands, Examples, Notes	0.15
type_declaration	Has effector.toml with [effector.interface] input/output/context	0.10
permission_alignment	Declared permissions match detected behavior (via effector-audit)	0.10
description_quality	Length, specificity, starts with verb, avoids vague language	0.05
install_completeness	At least one install method with all required fields	0.05

Functional Quality (requires execution sandbox)

Metric	What it checks	Weight
prerequisite_resolution	All declared requires.bins and requires.env resolve	0.10
invocation_success	Skill produces parseable output for a reference input	0.15
output_type_match	Actual output matches declared output type shape	0.10
error_handling	Graceful behavior on invalid input (no crash, clear message)	0.05

Composite Score

score = Σ(metric_score × weight)

Scale: 0.0 (broken) → 1.0 (production-ready). Thresholds:

Grade	Range	Meaning
A	0.85–1.00	Production-ready, publish to ClawHub
B	0.70–0.84	Functional, needs polish
C	0.50–0.69	Partially working, significant gaps
D	0.25–0.49	Fundamentally broken
F	0.00–0.24	Non-functional

Eval File Format

Evals are YAML files describing expected behavior:

# evals/linear.eval.yml
skill: linear
version: ">=1.0.0"

prerequisites:
  env:
    - LINEAR_API_KEY
  bins:
    - curl
    - jq

cases:
  - name: list-open-issues
    input: "What are my open Linear issues?"
    expect:
      output_type: JSON
      contains_fields: ["id", "title", "state"]
      no_error: true

  - name: create-issue
    input: "Create a Linear issue titled 'Test from skill-eval'"
    expect:
      output_type: JSON
      contains_fields: ["id", "identifier"]
      no_error: true
    teardown: "Delete the created issue"

  - name: invalid-key
    input: "List my issues"
    env_override:
      LINEAR_API_KEY: "invalid_key"
    expect:
      no_crash: true
      error_message_contains: ["unauthorized", "401", "invalid"]

scoring:
  pass_threshold: 0.70
  weights:
    invocation_success: 0.4
    output_type_match: 0.3
    error_handling: 0.2
    prerequisite_resolution: 0.1

Directory Structure

skill-eval/
├── src/
│   ├── runner.js          # Eval execution engine
│   ├── scorer.js          # Metric computation + grading
│   ├── reporter.js        # Output formatting (terminal, JSON, markdown)
│   └── static-analyzer.js # Structural quality checks (no execution)
├── evals/
│   ├── linear.eval.yml    # Reference eval for linear-skill
│   └── README.md          # How to write evals
├── fixtures/
│   ├── passing-skill/     # A skill that scores A
│   └── failing-skill/     # A skill that scores F (for testing the framework)
├── scripts/
│   └── run-eval.js        # CLI entry point
├── package.json
└── README.md

Usage

# Evaluate a single skill (structural only — no execution)
npx skill-eval ./path/to/skill --static-only

# Evaluate with execution (requires sandbox + prerequisites)
npx skill-eval ./path/to/skill --eval evals/linear.eval.yml

# Evaluate all skills in a directory
npx skill-eval ./skills/ --static-only --report markdown > report.md

# JSON output for CI
npx skill-eval ./path/to/skill --format json

Output Example

skill-eval v0.1.0 — linear-skill

Structural Quality
  ✓ frontmatter_completeness    1.00  (all fields present)
  ✓ section_coverage            1.00  (7/7 sections)
  ✓ type_declaration            1.00  (effector.toml with typed interface)
  ✓ permission_alignment        1.00  (no drift detected)
  ✓ description_quality         0.90  (good length, specific)
  ✓ install_completeness        1.00  (manual install with steps)

Functional Quality
  ✓ prerequisite_resolution     1.00  (curl, jq found; LINEAR_API_KEY set)
  ✓ invocation_success          1.00  (3/3 cases passed)
  ✓ output_type_match           1.00  (JSON output matches declaration)
  ✓ error_handling              0.80  (graceful on invalid key, no crash)

──────────────────────────────────
Score: 0.97 / 1.00  →  Grade A
──────────────────────────────────

Integration with effectorHQ

• skill-lint checks syntax and structure → skill-eval checks behavior and quality
• effector-audit checks permission drift → skill-eval uses that as one metric
• effector-types defines the type vocabulary → skill-eval checks output against declared types
• clawhub-analysis provides corpus baselines → skill-eval grades against ecosystem norms

Roadmap

• v0.1.0 — Static analyzer + reference eval for linear-skill
• v0.2.0 — Execution sandbox (Docker-based) + functional metrics
• v0.3.0 — CI integration (skill-eval-action for GitHub Actions)
• v0.4.0 — Batch mode for ClawHub-wide audits
• v1.0.0 — Stable API, published to npm as @effectorhq/skill-eval

Prior Art

• mgechev/skill-eval — Skill evaluation methodology
• Anthropic Claude Code Skills — Skill authoring + evaluation patterns
• OpenClaw SKILL.md spec — The skill format this evaluates
• effectorHQ/clawhub-analysis — Empirical data backing this framework

---

MIT License — effectorHQ Contributors