Skip to content

Implement AWS Glue Schema Registry Integration #8

@valdo404

Description

@valdo404

AWS Glue Schema Registry Integration

Description

Implement AWS Glue Schema Registry integration for schema management and evolution in the S3 sink connector, ensuring compatibility with AWS analytics services.

Tasks

  • Implement AWS Glue Schema Registry client in Rust
  • Support schema registration and retrieval
  • Handle schema versioning and compatibility checks
  • Integrate with Parquet and Avro serialization
  • Support automatic schema registration for new topics
  • Implement schema evolution rules (backward, forward, full compatibility)
  • Add configuration options for schema registry settings

Technical Details

  • Use AWS SDK for Rust to interact with Glue Schema Registry
  • Implement caching for schema lookups to improve performance
  • Support all serialization formats (Avro, JSON, Protobuf)
  • Add comprehensive tests for schema evolution scenarios
  • Ensure proper error handling for schema compatibility issues

Acceptance Criteria

  • Schemas are correctly registered and retrieved from AWS Glue Schema Registry
  • Schema evolution works according to configured compatibility rules
  • Integration with Parquet and Avro serialization is seamless
  • Performance meets or exceeds Java implementation
  • All tests pass including schema evolution scenarios

Priority

High (Priority 4 in GAP analysis)

Complexity

High

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestfeature:aws-integrationFeatures related to AWS service integrationpriority:highHigh priority task that should be addressed in the next release

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions