Implement SignatureOptimizer

[obie]

Overview

Implement SignatureOptimizer that automatically improves signature field descriptions to enhance model understanding and output quality.

Description

SignatureOptimizer rewrites and refines the descriptions in signature fields to make them clearer and more effective for the language model. This can significantly improve output quality without changing the core logic.

Key Features to Implement

• Analyze current signature descriptions
• Generate improved descriptions using LLM
• Test improvements empirically
• Support for multi-field signatures
• Preserve semantic meaning while improving clarity

Implementation Requirements

1. Optimization Process

class SignatureOptimizer < Base
  def optimize_signature(signature, dataset, metric)
    current_score = evaluate_signature(signature, dataset, metric)
    
    signature.fields.each do |field_name, field|
      # Generate alternative descriptions
      candidates = generate_descriptions(field, examples: dataset)
      
      # Test each candidate
      candidates.each do |new_description|
        temp_field = field.dup
        temp_field.description = new_description
        
        score = evaluate_signature(signature, dataset, metric)
        
        if score > current_score
          field.description = new_description
          current_score = score
        end
      end
    end
    
    signature
  end
end

2. Description Generation

def generate_descriptions(field, examples:)
  prompt = <<~PROMPT
    Field: #{field.name}
    Type: #{field.type}
    Current description: #{field.description}
    
    Examples of expected values:
    #{examples.map { |e| e[field.name] }.uniq.first(5).join("\n")}
    
    Generate 5 alternative descriptions that are:
    1. Clear and specific
    2. Include format hints if applicable
    3. Mention constraints or expectations
    4. Use terminology the model understands
  PROMPT
  
  # Generate alternatives using LLM
end

3. Features

• Iterative refinement
• A/B testing of descriptions
• Context-aware descriptions
• Format hint injection
• Constraint specification

Example Usage

# Original signature
sig = Desiru::Signature.new("question -> answer")
sig.fields[:question].description = "A question"
sig.fields[:answer].description = "The answer"

# Optimize descriptions
optimizer = Desiru::Optimizers::SignatureOptimizer.new(
  iterations: 3,
  candidates_per_field: 5,
  test_size: 20
)

optimized_sig = optimizer.compile(sig, dataset, metric)
# Results in:
# question: "A specific question requiring factual response, may include context"
# answer: "Comprehensive answer addressing all aspects of the question, include reasoning"

Configuration Options

• iterations: Optimization rounds (default: 2)
• candidates_per_field: Descriptions to generate per field
• test_size: Examples to use for testing
• improvement_threshold: Minimum improvement to accept
• preserve_original: Keep original as fallback

Testing Requirements

• Test with various signature types
• Measure actual performance improvements
• Test that semantic meaning is preserved
• Handle edge cases (empty descriptions, complex types)

Expected Benefits

• 5-10% improvement in output quality
• Better model understanding of expectations
• Reduced need for manual description tuning
• Discovers effective description patterns

Priority

Medium - Useful optimization but not critical for basic functionality