Skip to content

Evaluation: STT Config #595

@AkhileshNegi

Description

@AkhileshNegi

Is your feature request related to a problem?

In the first cut of STT evaluation, runs are created with hardcoded parameters (run_name, dataset_id, models). There is no way to pass a standardized config, making it difficult to manage and reproduce evaluation runs with different configurations (e.g. model settings, evaluation parameters). Text evaluation already solves this with config_id + config_version.

Describe the solution you'd like

Add config support to STT evaluation, following the text evaluation pattern:

  • Accept config_id and config_version as input parameters in STTEvaluationRunCreate
  • Resolve the config blob using ConfigVersionCrud before starting the evaluation batch
  • Store config_id and config_version in the EvaluationRun record
  • Extract STT-specific parameters (models, evaluation settings, etc.) from the resolved config

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions