-
Notifications
You must be signed in to change notification settings - Fork 11
fix(server): reject empty string list evaluator values #121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+170
−5
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
b6ceccd
fix: reject empty string list evaluator values
lan17 dcd6bcd
test: align new regressions with given-when-then style
lan17 8e20d0e
test: complete given-when-then comments in server tests
lan17 319eede
fix: reject blank list evaluator values
lan17 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| """Tests for list evaluator.""" | ||
|
|
||
| import pytest | ||
| from pydantic import ValidationError | ||
|
|
||
| from agent_control_evaluators.list import ListEvaluator, ListEvaluatorConfig | ||
|
|
||
|
|
||
| class TestListEvaluatorConfig: | ||
| """Tests for list evaluator config validation.""" | ||
|
|
||
| def test_empty_string_value_rejected(self) -> None: | ||
| """Test that empty-string list entries are rejected at config validation time.""" | ||
| # Given: a list evaluator config with an empty-string value | ||
| # When: constructing the config model | ||
| with pytest.raises( | ||
| ValidationError, match="values must not contain empty or whitespace-only strings" | ||
| ): | ||
| ListEvaluatorConfig(values=[""]) | ||
| # Then: validation rejects the config (asserted by pytest) | ||
|
|
||
| def test_whitespace_only_value_rejected(self) -> None: | ||
| """Test that whitespace-only list entries are rejected at config validation time.""" | ||
| # Given: a list evaluator config with a whitespace-only value | ||
| # When: constructing the config model | ||
| with pytest.raises( | ||
| ValidationError, match="values must not contain empty or whitespace-only strings" | ||
| ): | ||
| ListEvaluatorConfig(values=[" "]) | ||
| # Then: validation rejects the config (asserted by pytest) | ||
|
|
||
|
|
||
| class TestListEvaluator: | ||
| """Tests for list evaluator runtime behavior.""" | ||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_legacy_empty_string_value_is_ignored_defensively(self) -> None: | ||
| """Test that legacy invalid configs do not compile into a match-all regex.""" | ||
| # Given: a legacy invalid config constructed without validation | ||
| config = ListEvaluatorConfig.model_construct( | ||
| values=[""], | ||
| logic="any", | ||
| match_on="match", | ||
| match_mode="contains", | ||
| case_sensitive=False, | ||
| ) | ||
| evaluator = ListEvaluator(config) | ||
|
|
||
| # When: evaluating normal text against the legacy config | ||
| result = await evaluator.evaluate("Tell me a joke") | ||
|
|
||
| # Then: the evaluator ignores the empty control values | ||
| assert result.matched is False | ||
| assert result.message == "Empty control values - control ignored" | ||
lan17 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_legacy_whitespace_only_value_is_ignored_defensively(self) -> None: | ||
| """Test that legacy whitespace-only configs do not compile into pathological regexes.""" | ||
| # Given: a legacy invalid config with a whitespace-only value | ||
| config = ListEvaluatorConfig.model_construct( | ||
| values=[" "], | ||
| logic="any", | ||
| match_on="match", | ||
| match_mode="contains", | ||
| case_sensitive=False, | ||
| ) | ||
| evaluator = ListEvaluator(config) | ||
|
|
||
| # When: evaluating normal text against the legacy config | ||
| result = await evaluator.evaluate("Tell me a joke") | ||
|
|
||
| # Then: the evaluator ignores the empty control values | ||
| assert result.matched is False | ||
| assert result.message == "Empty control values - control ignored" | ||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_legacy_empty_string_allowlist_does_not_block_all(self) -> None: | ||
| """Test that legacy invalid allowlist configs do not block all inputs.""" | ||
| # Given: a legacy invalid allowlist config constructed without validation | ||
| config = ListEvaluatorConfig.model_construct( | ||
| values=[""], | ||
| logic="any", | ||
| match_on="no_match", | ||
| match_mode="contains", | ||
| case_sensitive=False, | ||
| ) | ||
| evaluator = ListEvaluator(config) | ||
|
|
||
| # When: evaluating normal text against the legacy config | ||
| result = await evaluator.evaluate("legitimate_value") | ||
|
|
||
| # Then: the evaluator ignores the empty control values instead of blocking all input | ||
| assert result.matched is False | ||
| assert result.message == "Empty control values - control ignored" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.