testing framework for transformRequest+Response

[knjiang]

This PR adds testing framework for transformRequest and transformResponse. This is specifically an offline testing framework where we test lingua transformations against saved snapshots.

We use the existing cases and WASM bindings generated from previous PR transform_request, transform_response and validate_*_json to verify valid transforms and no regressions.

High level, my mental model is:

• coverage-report ensures internal consistency between the Universal model & all the providers
• transforms ensures external compatibility with OpenAPI schema validations during every transform and using the actual SDK post-transform.

Diagram:

  Phase 1: Capture (one-time, requires API keys)                                                                                
                                                                                                                                
  getCaseForProvider(caseName, source)                                                                                          
             │                                                                                                                  
             ▼                                                                                                                  
  ┌─────────────────────────────────────┐                                                                                       
  │   transformAndValidateRequest()     │                                                                                       
  │   • transform_request() ────────────┼──► validate_*_request()                                                               
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
  ┌─────────────────────────────────────┐                                                                                       
  │   callProvider(target, request)     │                                                                                       
  │   • openai.chat.completions.create()│                                                                                       
  │   • anthropic.messages.create()     │                                                                                       
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
           validate_*_response()                                                                                                
                     │                                                                                                          
                     ▼                                                                                                          
           writeFileSync(path, response)                                                                                        
           transforms/{src}_to_{tgt}/{case}.json                                                                                
                                                                                                                                
  Phase 2: Test (CI, no API calls)                                                                                              
                                                                                                                                
  getCaseForProvider(caseName, source)                                                                                          
             │                                                                                                                  
             ▼                                                                                                                  
  ┌─────────────────────────────────────┐                                                                                       
  │   transformAndValidateRequest()     │                                                                                       
  │   • transform_request() ────────────┼──► validate_*_request()                                                               
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
             toMatchSnapshot("request")                                                                                         
                     │                                                                                                          
                     ▼                                                                                                          
  ┌─────────────────────────────────────┐                                                                                       
  │   loadAndValidateResponse()         │                                                                                       
  │   • readFileSync(path) ─────────────┼──► validate_*_response()                                                              
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
  ┌─────────────────────────────────────┐                                                                                       
  │   transformResponseData()           │                                                                                       
  │   • transform_response() ───────────┼──► validate_*_response()                                                              
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
             toMatchSnapshot("response")                                                                                        
                                                                                                                                

[knjiang]

• Support google parameters  #75
• add google test cases #74
• add google roundtrip #70
• testing framework for transformRequest+Response #72 👈 (View in Graphite)
• lingua-wasm bindings for request/response #69
• Add universal param configs #61
• add anthropic messages parameter test cases #59
• add chat completion parameter test cases #58
• add openai responses parameter test cases #54
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[knjiang]

only error i found so far, this seems to be valid since we define in our case max_tokens = 100 with high reasoning.

idk if we want to throw our own error or something. https://github.com/braintrustdata/lingua/blob/main/payloads/cases/simple.ts#L146

[knjiang]

caught this from the validate_response_json js binding. responses require input_token_details and output_token_details

[knjiang]

temporary, i didn't want to explode the diff so doing this here.

[knjiang]

anthropic_to_chatcompletions means anthropic payload using a chat completions model - we save the actual chat completion response payload so we don't have to incur the LLM cost.

[remh]

we should check that the errorReason matches the actual reason.

[knjiang]

the reason listed in transform_errors.json is a semantic reason for the reader, not the actual error.

[remh]

do we need that if the list is currently empty?

[knjiang]

this is a remnant deleting.