Skip to content

feat(operators): add follow-up error scenarios for operator null-return behaviour #356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aepfli wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(operators): add follow-up error scenarios for operator null-return behaviour #356

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b487daa
test(operators): add follow-up error scenarios for operator null-retu…
aepfli Mar 30, 2026
604df92
fixup! test(operators): add follow-up error scenarios for operator nu…
toddbaert Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 100 additions & 9 deletions evaluator/flags/testkit-flags.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,29 +462,38 @@
},
"semver-invalid-version-flag": {
"state": "ENABLED",
"variants": { "match": "match", "no-match": "no-match", "fallback": "fallback" },
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"sem_ver": [{"var": "version"}, "=", "1.0.0"]
"if": [
{"sem_ver": [{"var": "version"}, "=", "1.0.0"]},
"true", "false"
]
}
},
"semver-invalid-operator-flag": {
"state": "ENABLED",
"variants": { "match": "match", "no-match": "no-match", "fallback": "fallback" },
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"sem_ver": [{"var": "version"}, "===", "1.0.0"]
"if": [
{"sem_ver": [{"var": "version"}, "===", "1.0.0"]},
"true", "false"
]
}
},
"fractional-null-bucket-key-flag": {
"state": "ENABLED",
"variants": { "one": "one", "two": "two", "fallback": "fallback" },
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"fractional": [
{"var": "missing_key"},
["one", 50],
["two", 50]
"if": [
{"fractional": [
{"var": "missing_key"},
["one", 50],
["two", 50]
]},
"true", "false"
]
}
},
Expand Down Expand Up @@ -583,6 +592,88 @@
"miss"
]
}
},
"starts-with-non-string-flag": {
"state": "ENABLED",
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"if": [
{"starts_with": [{"var": "num"}, "abc"]},
"true", "false"
]
}
},
"ends-with-non-string-flag": {
"state": "ENABLED",
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"if": [
{"ends_with": [{"var": "num"}, "xyz"]},
"true", "false"
]
}
},
"starts-with-wrong-args-flag": {
"state": "ENABLED",
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"if": [
{"starts_with": ["abc"]},
"true", "false"
]
}
},
"ends-with-wrong-args-flag": {
"state": "ENABLED",
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"if": [
{"ends_with": ["xyz"]},
"true", "false"
]
}
},
"fractional-zero-weights-flag": {
"state": "ENABLED",
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"if": [
{"fractional": [
{"var": "targetingKey"},
["one", 0],
["two", 0]
]},
"true", "false"
]
}
},
"fractional-negative-weight-flag": {
"state": "ENABLED",
"variants": { "one": "one", "two": "two", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"fractional": [
{"var": "targetingKey"},
["one", -50],
["two", 100]
]
}
},
"semver-wrong-args-flag": {
"state": "ENABLED",
"variants": { "true": "true", "false": "false", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"if": [
{"sem_ver": [{"var": "version"}, "="]},
"true", "false"
]
}
}
},
"$evaluators": {
Expand Down
21 changes: 21 additions & 0 deletions evaluator/gherkin/fractional.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,3 +195,24 @@ Feature: Evaluator fractional operator
And a String-flag with key "fractional-null-bucket-key-flag" and a fallback value "wrong"
When the flag was evaluated with details
Then the resolved details value should be "fallback"
And the reason should be "DEFAULT"

# Follow-up error scenarios from https://github.com/open-feature/flagd/issues/1874

@operator-errors
Scenario: fractional with all-zero bucket weights falls back to default variant
Given an evaluator
And a String-flag with key "fractional-zero-weights-flag" and a fallback value "wrong"
And a context containing a targeting key with value "any-user"
When the flag was evaluated with details
Then the resolved details value should be "fallback"
And the reason should be "DEFAULT"

@operator-errors
Scenario: fractional negative bucket weight is clamped to zero
# ["one", -50] is treated as ["one", 0]; "two" gets 100% of the weight
Given an evaluator
And a String-flag with key "fractional-negative-weight-flag" and a fallback value "wrong"
And a context containing a targeting key with value "any-user"
When the flag was evaluated with details
Then the resolved details value should be "two"
12 changes: 12 additions & 0 deletions evaluator/gherkin/semver.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ Feature: Evaluator semantic version operator
And a context containing a key "version", with type "String" and with value "<context_value>"
When the flag was evaluated with details
Then the resolved details value should be "fallback"
And the reason should be "DEFAULT"
Examples:
| key | context_value |
| semver-invalid-version-flag | not-a-version |
Expand Down Expand Up @@ -86,3 +87,14 @@ Feature: Evaluator semantic version operator
| 1.0.0 | match |
| 1.0.0+other | match |
| 2.0.0 | no-match |

# Follow-up error scenarios from https://github.com/open-feature/flagd/issues/1874

@operator-errors
Scenario: sem_ver returns null for wrong argument count
Given an evaluator
And a String-flag with key "semver-wrong-args-flag" and a fallback value "wrong"
And a context containing a key "version", with type "String" and with value "1.0.0"
When the flag was evaluated with details
Then the resolved details value should be "fallback"
And the reason should be "DEFAULT"
29 changes: 29 additions & 0 deletions evaluator/gherkin/string.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,32 @@ Feature: Evaluator string comparison operator
| uvwxyz | postfix |
| abcxyz | prefix |
| lmnopq | none |

# Follow-up error scenarios from https://github.com/open-feature/flagd/issues/1874
# starts_with and ends_with must return null (not false) on error so that the
# default variant is selected, rather than looking up a non-existent "false" variant.

@operator-errors
Scenario Outline: starts_with and ends_with return null for non-string input
Given an evaluator
And a String-flag with key "<key>" and a fallback value "wrong"
And a context containing a key "num", with type "Integer" and with value "123"
When the flag was evaluated with details
Then the resolved details value should be "fallback"
And the reason should be "DEFAULT"
Examples:
| key |
| starts-with-non-string-flag |
| ends-with-non-string-flag |

@operator-errors
Scenario Outline: starts_with and ends_with return null for wrong argument count
Given an evaluator
And a String-flag with key "<key>" and a fallback value "wrong"
When the flag was evaluated with details
Then the resolved details value should be "fallback"
And the reason should be "DEFAULT"
Examples:
| key |
| starts-with-wrong-args-flag |
| ends-with-wrong-args-flag |
137 changes: 126 additions & 11 deletions flags/edge-case-flags.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,40 +56,155 @@
"semver-invalid-version-flag": {
"state": "ENABLED",
"variants": {
"match": "match",
"no-match": "no-match",
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"sem_ver": [{"var": "version"}, "=", "1.0.0"]
"if": [
{"sem_ver": [{"var": "version"}, "=", "1.0.0"]},
"true", "false"
]
}
},
"semver-invalid-operator-flag": {
"state": "ENABLED",
"variants": {
"match": "match",
"no-match": "no-match",
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"sem_ver": [{"var": "version"}, "===", "1.0.0"]
"if": [
{"sem_ver": [{"var": "version"}, "===", "1.0.0"]},
"true", "false"
]
}
},
"fractional-null-bucket-key-flag": {
"state": "ENABLED",
"variants": {
"one": "one",
"two": "two",
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"if": [
{"fractional": [
{"var": "missing_key"},
["one", 50],
["two", 50]
]},
"true", "false"
]
}
},
"starts-with-non-string-flag": {
"state": "ENABLED",
"variants": {
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"if": [
{"starts_with": [{"var": "num"}, "abc"]},
"true", "false"
]
}
},
"ends-with-non-string-flag": {
"state": "ENABLED",
"variants": {
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"if": [
{"ends_with": [{"var": "num"}, "xyz"]},
"true", "false"
]
}
},
"starts-with-wrong-args-flag": {
"state": "ENABLED",
"variants": {
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"if": [
{"starts_with": ["abc"]},
"true", "false"
]
}
},
"ends-with-wrong-args-flag": {
"state": "ENABLED",
"variants": {
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"if": [
{"ends_with": ["xyz"]},
"true", "false"
]
}
},
"fractional-zero-weights-flag": {
"state": "ENABLED",
"variants": {
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"if": [
{"fractional": [
{"var": "targetingKey"},
["one", 0],
["two", 0]
]},
"true", "false"
]
}
},
"fractional-negative-weight-flag": {
"state": "ENABLED",
"variants": { "one": "one", "two": "two", "fallback": "fallback" },
"defaultVariant": "fallback",
"targeting": {
"fractional": [
{"var": "missing_key"},
["one", 50],
["two", 50]
{"var": "targetingKey"},
["one", -50],
["two", 100]
]
}
},
"semver-wrong-args-flag": {
"state": "ENABLED",
"variants": {
"true": "true",
"false": "false",
"fallback": "fallback"
},
"defaultVariant": "fallback",
"targeting": {
"if": [
{"sem_ver": [{"var": "version"}, "="]},
"true", "false"
]
}
}
Expand Down
Loading
Loading