Implement AWS Glue Catalog Integration

[valdo404]

AWS Glue Catalog Integration

Description

Implement AWS Glue Data Catalog integration for the S3 sink connector to enable seamless table management and querying with AWS analytics services like Athena and EMR.

Tasks

• Implement AWS Glue Data Catalog client in Rust
• Support table creation and updates
• Handle partition registration directly from the connector
• Manage schema evolution in the catalog
• Add configuration options for Glue Catalog integration
• Support table properties and metadata management
• Implement efficient batch operations for partition registration

Technical Details

• Use AWS SDK for Rust to interact with Glue Data Catalog
• Implement caching for catalog operations to improve performance
• Support all Glue Data Catalog features relevant to S3 data
• Add comprehensive tests for catalog operations
• Ensure proper error handling for catalog API failures

Acceptance Criteria

• Tables are correctly created and updated in AWS Glue Data Catalog
• Partitions are properly registered without external crawlers
• Schema evolution is handled correctly in the catalog
• Performance meets or exceeds Java implementation
• All tests pass including edge cases like schema changes

Priority

Medium (Priority 8 in GAP analysis)

Complexity

Medium