Skip to content

Commit db9d03f

Browse files
aepfliCopilot
aepfli
and
Copilot
committed
test(operators): add follow-up error scenarios for operator null-return behaviour
Covers the remaining cases from open-feature/flagd#1874 identified in the review of #342: - starts_with / ends_with: non-string first argument → must return null - starts_with / ends_with: wrong argument count → must return null - sem_ver: wrong argument count (missing target version) → must return null - fractional: all-zero bucket weights (no bucket matched) → must return null - fractional: negative bucket weight clamped to zero → "one" gets effective weight 0, "two" gets 100% of the weight All scenarios use a bare operator as the targeting expression (not wrapped in 'if') so a null result selects the defaultVariant directly. Scenarios are tagged @operator-errors and mirrored in both the SDK-level gherkin (gherkin/targeting.feature) and the evaluator-level gherkin files (evaluator/gherkin/{string,semver,fractional}.feature) with matching flag definitions added to both flags/edge-case-flags.json and evaluator/flags/testkit-flags.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Simon Schrottner <simon.schrottner@dynatrace.com>
1 parent 2684a3e commit db9d03f

File tree

6 files changed

+230
-0
lines changed

6 files changed

+230
-0
lines changed

‎evaluator/flags/testkit-flags.json‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -446,6 +446,70 @@
446446
"small"
447447
]
448448
}
449+
},
450+
"starts-with-non-string-flag": {
451+
"state": "ENABLED",
452+
"variants": { "fallback": "fallback" },
453+
"defaultVariant": "fallback",
454+
"targeting": {
455+
"starts_with": [{"var": "num"}, "abc"]
456+
}
457+
},
458+
"ends-with-non-string-flag": {
459+
"state": "ENABLED",
460+
"variants": { "fallback": "fallback" },
461+
"defaultVariant": "fallback",
462+
"targeting": {
463+
"ends_with": [{"var": "num"}, "xyz"]
464+
}
465+
},
466+
"starts-with-wrong-args-flag": {
467+
"state": "ENABLED",
468+
"variants": { "fallback": "fallback" },
469+
"defaultVariant": "fallback",
470+
"targeting": {
471+
"starts_with": ["abc"]
472+
}
473+
},
474+
"ends-with-wrong-args-flag": {
475+
"state": "ENABLED",
476+
"variants": { "fallback": "fallback" },
477+
"defaultVariant": "fallback",
478+
"targeting": {
479+
"ends_with": ["xyz"]
480+
}
481+
},
482+
"fractional-zero-weights-flag": {
483+
"state": "ENABLED",
484+
"variants": { "one": "one", "two": "two", "fallback": "fallback" },
485+
"defaultVariant": "fallback",
486+
"targeting": {
487+
"fractional": [
488+
{"var": "targetingKey"},
489+
["one", 0],
490+
["two", 0]
491+
]
492+
}
493+
},
494+
"fractional-negative-weight-flag": {
495+
"state": "ENABLED",
496+
"variants": { "one": "one", "two": "two", "fallback": "fallback" },
497+
"defaultVariant": "fallback",
498+
"targeting": {
499+
"fractional": [
500+
{"var": "targetingKey"},
501+
["one", -50],
502+
["two", 100]
503+
]
504+
}
505+
},
506+
"semver-wrong-args-flag": {
507+
"state": "ENABLED",
508+
"variants": { "fallback": "fallback" },
509+
"defaultVariant": "fallback",
510+
"targeting": {
511+
"sem_ver": [{"var": "version"}, "="]
512+
}
449513
}
450514
},
451515
"$evaluators": {

‎evaluator/gherkin/fractional.feature‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,3 +181,22 @@ Feature: Evaluator fractional operator
181181
And a context containing a targeting key with value "some-targeting-key"
182182
When the flag was evaluated with details
183183
Then the resolved details value should be "ones"
184+
185+
# Follow-up error scenarios from https://github.com/open-feature/flagd/issues/1874
186+
187+
@operator-errors
188+
Scenario: fractional with all-zero bucket weights falls back to default variant
189+
Given an evaluator
190+
And a String-flag with key "fractional-zero-weights-flag" and a fallback value "wrong"
191+
And a context containing a targeting key with value "any-user"
192+
When the flag was evaluated with details
193+
Then the resolved details value should be "fallback"
194+
195+
@operator-errors
196+
Scenario: fractional negative bucket weight is clamped to zero
197+
# ["one", -50] is treated as ["one", 0]; "two" gets 100% of the weight
198+
Given an evaluator
199+
And a String-flag with key "fractional-negative-weight-flag" and a fallback value "wrong"
200+
And a context containing a targeting key with value "any-user"
201+
When the flag was evaluated with details
202+
Then the resolved details value should be "two"

‎evaluator/gherkin/semver.feature‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,13 @@ Feature: Evaluator semantic version operator
3131
| 3.0.1 | minor |
3232
| 3.1.0 | major |
3333
| 4.0.0 | none |
34+
35+
# Follow-up error scenarios from https://github.com/open-feature/flagd/issues/1874
36+
37+
@operator-errors
38+
Scenario: sem_ver returns null for wrong argument count
39+
Given an evaluator
40+
And a String-flag with key "semver-wrong-args-flag" and a fallback value "wrong"
41+
And a context containing a key "version", with type "String" and with value "1.0.0"
42+
When the flag was evaluated with details
43+
Then the resolved details value should be "fallback"

‎evaluator/gherkin/string.feature‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,30 @@ Feature: Evaluator string comparison operator
1616
| uvwxyz | postfix |
1717
| abcxyz | prefix |
1818
| lmnopq | none |
19+
20+
# Follow-up error scenarios from https://github.com/open-feature/flagd/issues/1874
21+
# starts_with and ends_with must return null (not false) on error so that the
22+
# default variant is selected, rather than looking up a non-existent "false" variant.
23+
24+
@operator-errors
25+
Scenario Outline: starts_with and ends_with return null for non-string input
26+
Given an evaluator
27+
And a String-flag with key "<key>" and a fallback value "wrong"
28+
And a context containing a key "num", with type "Integer" and with value "123"
29+
When the flag was evaluated with details
30+
Then the resolved details value should be "fallback"
31+
Examples:
32+
| key |
33+
| starts-with-non-string-flag |
34+
| ends-with-non-string-flag |
35+
36+
@operator-errors
37+
Scenario Outline: starts_with and ends_with return null for wrong argument count
38+
Given an evaluator
39+
And a String-flag with key "<key>" and a fallback value "wrong"
40+
When the flag was evaluated with details
41+
Then the resolved details value should be "fallback"
42+
Examples:
43+
| key |
44+
| starts-with-wrong-args-flag |
45+
| ends-with-wrong-args-flag |

‎flags/edge-case-flags.json‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,70 @@
5252
},
5353
"defaultVariant": "false",
5454
"targeting": {}
55+
},
56+
"starts-with-non-string-flag": {
57+
"state": "ENABLED",
58+
"variants": { "fallback": "fallback" },
59+
"defaultVariant": "fallback",
60+
"targeting": {
61+
"starts_with": [{"var": "num"}, "abc"]
62+
}
63+
},
64+
"ends-with-non-string-flag": {
65+
"state": "ENABLED",
66+
"variants": { "fallback": "fallback" },
67+
"defaultVariant": "fallback",
68+
"targeting": {
69+
"ends_with": [{"var": "num"}, "xyz"]
70+
}
71+
},
72+
"starts-with-wrong-args-flag": {
73+
"state": "ENABLED",
74+
"variants": { "fallback": "fallback" },
75+
"defaultVariant": "fallback",
76+
"targeting": {
77+
"starts_with": ["abc"]
78+
}
79+
},
80+
"ends-with-wrong-args-flag": {
81+
"state": "ENABLED",
82+
"variants": { "fallback": "fallback" },
83+
"defaultVariant": "fallback",
84+
"targeting": {
85+
"ends_with": ["xyz"]
86+
}
87+
},
88+
"fractional-zero-weights-flag": {
89+
"state": "ENABLED",
90+
"variants": { "one": "one", "two": "two", "fallback": "fallback" },
91+
"defaultVariant": "fallback",
92+
"targeting": {
93+
"fractional": [
94+
{"var": "targetingKey"},
95+
["one", 0],
96+
["two", 0]
97+
]
98+
}
99+
},
100+
"fractional-negative-weight-flag": {
101+
"state": "ENABLED",
102+
"variants": { "one": "one", "two": "two", "fallback": "fallback" },
103+
"defaultVariant": "fallback",
104+
"targeting": {
105+
"fractional": [
106+
{"var": "targetingKey"},
107+
["one", -50],
108+
["two", 100]
109+
]
110+
}
111+
},
112+
"semver-wrong-args-flag": {
113+
"state": "ENABLED",
114+
"variants": { "fallback": "fallback" },
115+
"defaultVariant": "fallback",
116+
"targeting": {
117+
"sem_ver": [{"var": "version"}, "="]
118+
}
55119
}
56120
}
57121
}

‎gherkin/targeting.feature‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,3 +273,49 @@ Feature: Targeting rules
273273
| missing-variant-targeting-flag | 3 | GENERAL |
274274
| non-string-variant-targeting-flag | 2 | |
275275
| empty-targeting-flag | 1 | |
276+
277+
# Follow-up error scenarios from https://github.com/open-feature/flagd/issues/1874
278+
# Operators must return null (not false) on error so the default variant is selected.
279+
280+
@operator-errors @string
281+
Scenario Outline: starts_with and ends_with return null for non-string input
282+
Given a String-flag with key "<key>" and a default value "wrong"
283+
And a context containing a key "num", with type "Integer" and with value "123"
284+
When the flag was evaluated with details
285+
Then the resolved details value should be "fallback"
286+
Examples:
287+
| key |
288+
| starts-with-non-string-flag |
289+
| ends-with-non-string-flag |
290+
291+
@operator-errors @string
292+
Scenario Outline: starts_with and ends_with return null for wrong argument count
293+
Given a String-flag with key "<key>" and a default value "wrong"
294+
When the flag was evaluated with details
295+
Then the resolved details value should be "fallback"
296+
Examples:
297+
| key |
298+
| starts-with-wrong-args-flag |
299+
| ends-with-wrong-args-flag |
300+
301+
@operator-errors @semver
302+
Scenario: sem_ver returns null for wrong argument count
303+
Given a String-flag with key "semver-wrong-args-flag" and a default value "wrong"
304+
And a context containing a key "version", with type "String" and with value "1.0.0"
305+
When the flag was evaluated with details
306+
Then the resolved details value should be "fallback"
307+
308+
@operator-errors @fractional
309+
Scenario: fractional with all-zero bucket weights falls back to default variant
310+
Given a String-flag with key "fractional-zero-weights-flag" and a default value "wrong"
311+
And a context containing a targeting key with value "any-user"
312+
When the flag was evaluated with details
313+
Then the resolved details value should be "fallback"
314+
315+
@operator-errors @fractional
316+
Scenario: fractional negative bucket weight is clamped to zero
317+
# ["one", -50] is treated as ["one", 0]; "two" gets 100% of the weight
318+
Given a String-flag with key "fractional-negative-weight-flag" and a default value "wrong"
319+
And a context containing a targeting key with value "any-user"
320+
When the flag was evaluated with details
321+
Then the resolved details value should be "two"

0 commit comments

Comments
 (0)