feat: LLM Optimizer #9
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced `evaluation_mode` and `evaluation_criteria` fields in the prompts table and related schemas. - Updated frontend components to support selection and input of evaluation mode and criteria. - Modified API endpoints to handle new fields during prompt creation and import. - Implemented LLM evaluation logic in the test runner, allowing for dynamic evaluation based on selected mode. - Enhanced validation to ensure criteria are provided when using LLM evaluation mode.
- Updated the API to accept an optional `evaluationModel` parameter in the test run request. - Enhanced the frontend to allow users to select an evaluation model when running tests. - Implemented logic to handle the evaluation model in the test runner, ensuring proper evaluation during LLM evaluation mode. - Added validation for the evaluation model selection in the test run schema.
- Implemented a new function to format and display the evaluation model in the test run results. - Updated the TestResults interface to include an optional evaluationModel property. - Enhanced the test runner to pass the evaluation model information when running tests. - Modified the frontend to conditionally render the evaluation model label for completed test runs.
- Added conditional rendering for the evaluation model label in the test run results. - Removed redundant evaluation model display logic from previous run information. - Ensured the evaluation model is displayed only when available, improving clarity in the UI.
- Introduced optional parameters for optimization in the test run API, including `optimizationMaxIterations`, `optimizationThreshold`, and `optimizationModel`. - Updated the frontend to allow users to configure optimization settings, including UI elements for selecting optimization models and setting thresholds. - Enhanced the test runner to handle optimization logic, including tracking optimization history and integrating with LLM evaluation. - Added validation for new optimization parameters in the test run schema.
Collaborator
Author
|
check #10 instead |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This branch is based on #8, check this after #8 gets merged