feat: Add AWS S3 adapter for AFS

[hadv]

Overview

This PR implements a comprehensive AWS S3 adapter for NebulaStore's Abstract File System (AFS), migrating and adapting the Eclipse Store AWS S3 adapter to work with NebulaStore's .NET architecture.

✅ Build Status

Latest Updates:

• ✅ Fixed compilation errors with path validator interface
• ✅ Corrected IAwsS3PathValidator to extend IAfsPathValidator
• ✅ Updated validation calls to use proper path.Validate(validator) pattern
• ✅ Added missing using statements
• ✅ Updated package versions for better compatibility
• ✅ Resolved AWS SDK package version resolution issues
• ✅ Fixed AWS SDK API compatibility and code analysis issues

Features

Core Implementation

• AwsS3Connector: Full implementation of IBlobStoreConnector interface
• AwsS3PathValidator: S3-specific path validation following AWS naming conventions
• AwsS3Configuration: Flexible configuration system for S3 connections

AWS S3 Support

• ✅ Multi-region support (all AWS regions)
• ✅ S3-compatible services support (MinIO, LocalStack, etc.)
• ✅ Both path-style and virtual-hosted-style addressing
• ✅ HTTPS/HTTP support for different environments
• ✅ Configurable timeouts and retry policies
• ✅ Optional caching layer for improved performance

File Operations

• ✅ Large file support with automatic blob splitting
• ✅ Thread-safe operations
• ✅ Full CRUD operations (Create, Read, Update, Delete)
• ✅ Directory operations and traversal
• ✅ File operations (move, copy, truncate)
• ✅ Proper resource disposal and error handling

Files Added

Core Implementation

• afs/aws/s3/NebulaStore.Afs.Aws.S3.csproj - Main project file
• afs/aws/s3/src/AwsS3Connector.cs - Main S3 connector implementation
• afs/aws/s3/src/AwsS3PathValidator.cs - S3 path validation
• afs/aws/s3/src/AwsS3Configuration.cs - Configuration management

Testing

• afs/aws/s3/test/NebulaStore.Afs.Aws.S3.Tests.csproj - Test project
• afs/aws/s3/test/AwsS3ConnectorTests.cs - Comprehensive unit tests with mocking

Documentation & Examples

• afs/aws/s3/README.md - Detailed documentation with usage examples
• examples/AwsS3Example.cs - Complete usage examples for different scenarios

Files Modified

• NebulaStore.sln - Added new AWS S3 projects to solution
• afs/README.md - Updated with AWS S3 adapter documentation

Usage Examples

Basic Usage

// Create S3 client
var s3Client = new AmazonS3Client("access-key", "secret-key", RegionEndpoint.USEast1);

// Create S3 configuration
var s3Config = AwsS3Configuration.New()
    .SetCredentials("access-key", "secret-key")
    .SetRegion(RegionEndpoint.USEast1)
    .SetUseCache(true);

// Create connector and file system
using var connector = AwsS3Connector.New(s3Client, s3Config);
using var fileSystem = BlobStoreFileSystem.New(connector);

Advanced Configuration

// Advanced S3 configuration
var s3Config = AwsS3Configuration.New()
    .SetCredentials("access-key", "secret-key")
    .SetRegion(RegionEndpoint.USWest2)
    .SetUseCache(true)
    .SetTimeout(60000) // 60 seconds
    .SetMaxRetryAttempts(5)
    .SetForcePathStyle(false)
    .SetUseHttps(true);

S3-Compatible Services (MinIO)

// MinIO configuration
var minioConfig = AwsS3Configuration.New()
    .SetCredentials("minio-access-key", "minio-secret-key")
    .SetServiceUrl("http://localhost:9000")
    .SetForcePathStyle(true)
    .SetUseHttps(false);

Architecture

The implementation follows the same patterns as the existing NebulaStore AFS adapters:

NebulaStore Application
    ↓
BlobStoreFileSystem
    ↓
AwsS3Connector (implements IBlobStoreConnector)
    ↓
AWS S3 Client → S3 Bucket → Blob Objects

Testing

Unit Tests

• Comprehensive test coverage with Moq for S3 client mocking
• Tests for all connector operations
• Error handling and edge case testing

Integration Testing

To test with real AWS S3:

1. Set up AWS credentials
2. Create an S3 bucket
3. Run the examples in examples/AwsS3Example.cs

Dependencies

• AWSSDK.S3 (version 3.7.401) - AWS SDK for .NET S3 operations
• Existing NebulaStore AFS infrastructure

Validation

The implementation includes comprehensive S3 bucket name validation according to AWS rules:

• Length between 3-63 characters
• Lowercase letters, numbers, periods, and dashes only
• Must start with letter or number
• Cannot end with dash
• No consecutive periods
• No IP address format
• Cannot start with 'xn--'

Performance Considerations

• Optional caching layer for improved performance
• Efficient blob numbering for large file handling
• Configurable timeouts and retry policies
• Thread-safe operations

Security

• Support for IAM roles and temporary credentials
• HTTPS support by default
• Proper credential management
• No hardcoded secrets in code

Breaking Changes

None - this is a new feature addition that doesn't affect existing functionality.

Recent Fixes

Build Issues Resolved ✅

Commit: 4f8b338 - Fixed path validator interface usage:

• Corrected IAwsS3PathValidator to extend IAfsPathValidator instead of non-existent BlobStorePath.IValidator
• Updated Validate method to accept IAfsPath parameter with proper type checking
• Fixed all validation calls in AwsS3Connector to use path.Validate(validator) pattern

Commit: 029819e - Added missing using statement:

• Added NebulaStore.Afs.Blobstore.Types using statement to ensure proper interface resolution

Commit: 14ad272 - Updated package versions for better compatibility:

• Downgraded AWSSDK.S3 from 3.7.400.44 to 3.7.300 for better stability
• Updated test packages to more stable versions

Commit: 5511b92 - Resolved AWS SDK package version issues:

• Updated AWSSDK.S3 version to 3.7.401 (the version that was actually resolved by NuGet)
• Added NU1603 to WarningsNotAsErrors to prevent package version resolution warnings from failing the build
• Fixed the specific error: NebulaStore.Afs.Aws.S3 depends on AWSSDK.S3 (>= 3.7.400.44) but AWSSDK.S3 3.7.400.44 was not found. AWSSDK.S3 3.7.401 was resolved instead.

Commit: 5047faf - Fixed AWS SDK API compatibility and code analysis issues:

• Removed ContentLength property from PutObjectRequest (not available in current AWS SDK version)
• Fixed CA2022 code analysis warning by replacing inexact Stream.Read() with proper read loop pattern
• Improved stream reading reliability by ensuring all bytes are read from the response stream
• Fixed compilation errors:
  • CS0117: 'PutObjectRequest' does not contain a definition for 'ContentLength'
  • CA2022: Avoid inexact read with 'System.IO.Stream.Read(byte[], int, int)'

Error Analysis 🔍

Latest Issues Resolved:

1. AWS SDK API Changes: The ContentLength property was removed from PutObjectRequest in newer AWS SDK versions. The SDK now automatically determines content length from the input stream.

2. Code Analysis Rules: .NET code analysis flagged the use of Stream.Read() as potentially unsafe because it doesn't guarantee reading the requested number of bytes. Replaced with a proper read loop that continues until all bytes are read or end of stream is reached.

Checklist

• Implementation follows NebulaStore coding standards
• Comprehensive unit tests included
• Documentation and examples provided
• Solution file updated
• No breaking changes to existing code
• Thread-safe implementation
• Proper error handling and resource disposal
• AWS S3 naming conventions validated
• Build compilation errors fixed
• Proper interface inheritance implemented
• All using statements added
• Package versions updated for compatibility
• AWS SDK package version resolution issues fixed
• AWS SDK API compatibility issues resolved
• Code analysis warnings addressed

Related Issues

This implements the AWS S3 adapter migration as requested, bringing Eclipse Store's AWS S3 functionality to NebulaStore's .NET ecosystem.

---

Pull Request opened by Augment Code with guidance from the PR author