[EPIC] Document - Multi-Modal Quality Evaluation and Generation #127
Description
Problem Statement
Strands-evals SDK needs to support quality evaluation for multi-modal content (images, documents, audio). Currently, the SDK can evaluate tool selection and parameters, but cannot assess the actual quality of generated or processed multi-modal outputs.
1. Current Capabilities
- Evaluates text-based outputs using LLM-as-a-judge
- Verifies tool selection and parameter accuracy
- Analyzes agent trajectories and interactions
- Captures multi-modal tool calls in traces (but doesn't evaluate the content)
2. Missing Capabilities
- Modal Quality Assessment - Cannot evaluate if generated images match prompts, if documents are correctly processed, or if audio transcriptions are accurate
- Tool-Enabled Evaluation - Evaluators cannot use tools (vision models, document parsers, audio analyzers) to perform automated quality checks
- Multi-Modal Experiment Generation - ExperimentGenerator cannot create test cases for image generation, document processing, or audio tasks
Possible Proposed Solutions
1. Allow evaluators to use tools
Allow evaluators to invoke tools during evaluation (vision analysis, document parsing, audio processing).
2. New Evaluators:
- ImageQualityEvaluator - Assesses generated/processed images for prompt adherence, visual quality, and technical correctness
- DocumentQualityEvaluator - Validates document extraction accuracy, structure preservation, and completeness
- AudioQualityEvaluator - Evaluates transcription accuracy, audio quality, and speech clarity
Tool Integration:
3. Experiment Generation:
Extend ExperimentGenerator to create multi-modal test cases with expected visual/document/audio outputs
Generate rubrics that include content-type-specific quality criteria
Use Case
N/A
Alternatives Solutions
No response
Additional Context
No response