Parent Issue
This is a sub-issue of open-feature/flagd#1904.
Summary
The flagd-testbed gherkin suite needs to be enhanced to cover edge cases and error handling inconsistencies identified across the in-process provider implementations. The testbed already has some error case coverage but needs significant expansion to serve as the source of truth for consistent behavior.
Background
Several consistency bugs have been identified across the SDK implementations (Go, Java, JS/TS, .NET, Python):
Note: fractional bucketing formula differences (open-feature/flagd#1872) will be addressed separately by the high-resolution fractional bucketing work in open-feature/flagd#1903.
Required Test Additions
sem_ver operator (flagd#1873)
v-prefixed versions: "v1.2.3" should be accepted and treated as "1.2.3"
- Partial versions: decide and test whether
"1.2" / "1" are accepted (padded to X.Y.0 / X.0.0) or rejected
- Build metadata ignored in comparisons (SemVer 2.0.0 Rule 10):
"1.0.0+build1" == "1.0.0+build2" → true
"1.0.0+build" == "1.0.0" → true
- Parse failure on invalid version strings returns
null
Error return standardization (flagd#1874)
All custom operators should return null on error (not false, not empty string, not exception). This ensures errors trigger a fallback to the default variant rather than taking the false branch of a conditional.
Test cases needed:
fractional with invalid args → null (default variant)
fractional no-bucket-match → null (default variant)
sem_ver with unparseable version → null (default variant)
sem_ver with unknown operator → null (default variant)
starts_with with wrong arg types → null (default variant)
ends_with with wrong arg types → null (default variant)
$evaluators/$ref (flagd#1875)
- Unresolved
$ref → clear error / PARSE_ERROR
- Whitespace variations in
$ref objects
- Nested
$ref resolution
Notes
These tests will serve as the conformance suite that each provider implementation must pass. Once this suite is enhanced, each provider can vet and fix its implementation against it.
Parent Issue
This is a sub-issue of open-feature/flagd#1904.
Summary
The flagd-testbed gherkin suite needs to be enhanced to cover edge cases and error handling inconsistencies identified across the in-process provider implementations. The testbed already has some error case coverage but needs significant expansion to serve as the source of truth for consistent behavior.
Background
Several consistency bugs have been identified across the SDK implementations (Go, Java, JS/TS, .NET, Python):
sem_verparsing and comparison inconsistencies$evaluators/$refresolution gapsRequired Test Additions
sem_veroperator (flagd#1873)v-prefixed versions:"v1.2.3"should be accepted and treated as"1.2.3""1.2"/"1"are accepted (padded toX.Y.0/X.0.0) or rejected"1.0.0+build1" == "1.0.0+build2"→true"1.0.0+build" == "1.0.0"→truenullError return standardization (flagd#1874)
All custom operators should return
nullon error (notfalse, not empty string, not exception). This ensures errors trigger a fallback to the default variant rather than taking the false branch of a conditional.Test cases needed:
fractionalwith invalid args →null(default variant)fractionalno-bucket-match →null(default variant)sem_verwith unparseable version →null(default variant)sem_verwith unknown operator →null(default variant)starts_withwith wrong arg types →null(default variant)ends_withwith wrong arg types →null(default variant)$evaluators/$ref(flagd#1875)$ref→ clear error / PARSE_ERROR$refobjects$refresolutionNotes
These tests will serve as the conformance suite that each provider implementation must pass. Once this suite is enhanced, each provider can vet and fix its implementation against it.