Skip to content

[FEATURE] Enhance gherkin suite for custom operator edge cases and error handling consistency #339

@toddbaert

Description

@toddbaert

Parent Issue

This is a sub-issue of open-feature/flagd#1904.

Summary

The flagd-testbed gherkin suite needs to be enhanced to cover edge cases and error handling inconsistencies identified across the in-process provider implementations. The testbed already has some error case coverage but needs significant expansion to serve as the source of truth for consistent behavior.

Background

Several consistency bugs have been identified across the SDK implementations (Go, Java, JS/TS, .NET, Python):

Note: fractional bucketing formula differences (open-feature/flagd#1872) will be addressed separately by the high-resolution fractional bucketing work in open-feature/flagd#1903.

Required Test Additions

sem_ver operator (flagd#1873)

  • v-prefixed versions: "v1.2.3" should be accepted and treated as "1.2.3"
  • Partial versions: decide and test whether "1.2" / "1" are accepted (padded to X.Y.0 / X.0.0) or rejected
  • Build metadata ignored in comparisons (SemVer 2.0.0 Rule 10):
    • "1.0.0+build1" == "1.0.0+build2"true
    • "1.0.0+build" == "1.0.0"true
  • Parse failure on invalid version strings returns null

Error return standardization (flagd#1874)

All custom operators should return null on error (not false, not empty string, not exception). This ensures errors trigger a fallback to the default variant rather than taking the false branch of a conditional.

Test cases needed:

  • fractional with invalid args → null (default variant)
  • fractional no-bucket-match → null (default variant)
  • sem_ver with unparseable version → null (default variant)
  • sem_ver with unknown operator → null (default variant)
  • starts_with with wrong arg types → null (default variant)
  • ends_with with wrong arg types → null (default variant)

$evaluators/$ref (flagd#1875)

  • Unresolved $ref → clear error / PARSE_ERROR
  • Whitespace variations in $ref objects
  • Nested $ref resolution

Notes

These tests will serve as the conformance suite that each provider implementation must pass. Once this suite is enhanced, each provider can vet and fix its implementation against it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Pending

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions