refactor!: remove deprecated Eval API and related APIs#5290
refactor!: remove deprecated Eval API and related APIs#5290leseb wants to merge 7 commits intollamastack:mainfrom
Conversation
…o, scoring, scoring_functions, benchmarks) Remove the Eval API and all connected APIs that were already marked as deprecated in the spec. This includes the datasets, datasetio, scoring, scoring_functions, and benchmarks APIs along with all their provider implementations, routing tables, routers, registry entries, distribution configs, and tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sébastien Han <seb@redhat.com>
meta-cla
bot
added
the
CLA Signed
✱ Stainless preview buildsThis PR will update the Edit this comment to update it. It will appear in the SDK's changelogs. ✅ llama-stack-client-node studio · conflict
✅ llama-stack-client-go studio · conflict
✅ llama-stack-client-openapi studio · code · diff
✅ llama-stack-client-python studio · conflict
This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push. |
|
This pull request has merge conflicts that must be resolved before it can be merged. @leseb please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
Signed-off-by: Sébastien Han <seb@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
|
This pull request has merge conflicts that must be resolved before it can be merged. @leseb please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
…nses rename Merge upstream/main into remove-eval-api branch, resolving conflicts caused by the agents-to-responses API rename (PR llamastack#5195) interacting with the eval API removal. This includes fixing distribution templates that still referenced the old "agents" API name, removing the stale DatasetPurpose import from template.py, cleaning up the stainless config to remove references to deleted eval/scoring/dataset endpoints, removing the now-orphaned nvidia datasetio test, and regenerating all OpenAPI specs and distribution configs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sébastien Han <seb@redhat.com>
Resolve merge conflicts from upstream/main. All modify/delete conflicts are resolved by keeping the deletions from this branch since the purpose of this branch is to remove the eval, scoring, datasetio, and benchmarks APIs. Content conflicts in datatypes.py files are resolved by excluding the eval-related classes that upstream added. Signed-off-by: Sébastien Han <seb@redhat.com>
Merge upstream/main into remove-eval-api branch. The Dell distribution was removed upstream, so the conflicting Dell files (config.yaml, dell.py, run-with-safety.yaml) are resolved by accepting the upstream deletion. Signed-off-by: Sebastien Han <seb@redhat.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sébastien Han <seb@redhat.com>
1 participant
Summary
Remove the Eval API and all connected APIs that are already marked deprecated in the spec:
/v1alpha/eval/(benchmarks, jobs, evaluations)/v1beta/datasets/(dataset CRUD)/v1beta/datasetio/(append-rows, iterrows)/v1/scoring-functions/(scoring function CRUD)These APIs have limited adoption and are better served by purpose-built tools (MLflow, eval frameworks). Removing them sharpens the project's focus on OpenAI-compatible agentic APIs.
What was removed
Stats
Test plan
Signed-off-by: Sebastien Han shan@redhat.com