Skip to content

Experiment: docs-only vs skill-guided agent — compare outcomes #112

@tomkis

Description

@tomkis

Is your feature request related to a problem? Please describe.
Evaluating whether skills can be replaced by documentation links. It's unclear if guiding an agent via docs alone produces equivalent results to providing a structured skill — this creates uncertainty about when a skill is actually necessary.

Describe the solution you'd like
Run a controlled experiment:

  1. Take an existing skill (e.g. the ADK wrapper skill)
  2. Run the same prompt twice — once with the skill, once with the agent pointed only at the documentation (no skill provided)
  3. Measure: does the agent find/use the docs on its own, or does it need to be directed? Are outcomes equivalent?

Describe alternatives you've considered

  • Always providing a skill regardless of whether docs alone would suffice
  • Relying on anecdotal observations without a structured comparison

Additional context
Result should either validate skills-as-doc-pointers or identify where structured skills add irreplaceable value.

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions