Skip to content

Implement Azure Data Lake Storage (ADLS) cloud connections#183

Open
Edwardvaneechoud wants to merge 4 commits intomainfrom
claude/implement-adls-connections-JhgiG
Open

Implement Azure Data Lake Storage (ADLS) cloud connections#183
Edwardvaneechoud wants to merge 4 commits intomainfrom
claude/implement-adls-connections-JhgiG

Conversation

@Edwardvaneechoud
Copy link
Owner

This commit adds comprehensive support for ADLS Gen2 storage with multiple
authentication methods, full read/write capabilities, and local testing support
via Azurite emulator.

Backend changes:

  • Add azure_sas_token field to cloud storage schemas and database models
  • Implement ADLS storage options for access_key, service_principal, and sas_token auth
  • Add ADLS path parsing utilities (az:// and abfs:// URI formats)
  • Add Azure Blob Storage client integration with file listing capabilities
  • Update cloud storage reader to support ADLS paths alongside S3
  • Store and retrieve ADLS secrets (account_key, client_secret, sas_token)

Worker changes:

  • Add ADLS storage options to worker models with secret decryption
  • Enable ADLS write operations for all file formats (Parquet, CSV, JSON, Delta)
  • Support all ADLS authentication methods in worker context

Frontend changes:

  • Enable ADLS storage type option in cloud connection UI
  • Add UI fields for Azure-specific credentials (account name, keys, tenant, etc.)
  • Add SAS token authentication support with password field visibility toggle
  • Add azure_sas_token to TypeScript type definitions and API conversions
  • Update form validation for ADLS connection requirements

Dependencies:

  • Add azure-storage-blob ^12.24.0 for Blob Storage SDK
  • Add azure-identity ^1.19.0 for service principal authentication
  • Add adlfs ^2025.1.0 for Polars integration

Testing infrastructure:

  • Create test_utils/adls module for Azurite-based testing
  • Add start_azurite and stop_azurite CLI commands
  • Implement test data generator with Parquet, CSV, and JSON samples
  • Add comprehensive README with usage examples and troubleshooting
  • Create managed_azurite context manager for integration tests

Supported features:

  • Multiple authentication methods: access key, service principal, SAS token
  • Read operations: single files and directory wildcards
  • Write operations: Parquet, CSV, JSON, Delta Lake formats
  • Path formats: both az:// and abfs:// URI schemes
  • Local testing with Azurite emulator (Docker-based)
  • Secure secret storage with per-user encryption

Files modified: 9 backend/frontend files
Files created: 5 test utility files

claude and others added 4 commits January 10, 2026 08:09
This commit adds comprehensive support for ADLS Gen2 storage with multiple
authentication methods, full read/write capabilities, and local testing support
via Azurite emulator.

Backend changes:
- Add azure_sas_token field to cloud storage schemas and database models
- Implement ADLS storage options for access_key, service_principal, and sas_token auth
- Add ADLS path parsing utilities (az:// and abfs:// URI formats)
- Add Azure Blob Storage client integration with file listing capabilities
- Update cloud storage reader to support ADLS paths alongside S3
- Store and retrieve ADLS secrets (account_key, client_secret, sas_token)

Worker changes:
- Add ADLS storage options to worker models with secret decryption
- Enable ADLS write operations for all file formats (Parquet, CSV, JSON, Delta)
- Support all ADLS authentication methods in worker context

Frontend changes:
- Enable ADLS storage type option in cloud connection UI
- Add UI fields for Azure-specific credentials (account name, keys, tenant, etc.)
- Add SAS token authentication support with password field visibility toggle
- Add azure_sas_token to TypeScript type definitions and API conversions
- Update form validation for ADLS connection requirements

Dependencies:
- Add azure-storage-blob ^12.24.0 for Blob Storage SDK
- Add azure-identity ^1.19.0 for service principal authentication
- Add adlfs ^2025.1.0 for Polars integration

Testing infrastructure:
- Create test_utils/adls module for Azurite-based testing
- Add start_azurite and stop_azurite CLI commands
- Implement test data generator with Parquet, CSV, and JSON samples
- Add comprehensive README with usage examples and troubleshooting
- Create managed_azurite context manager for integration tests

Supported features:
- Multiple authentication methods: access key, service principal, SAS token
- Read operations: single files and directory wildcards
- Write operations: Parquet, CSV, JSON, Delta Lake formats
- Path formats: both az:// and abfs:// URI schemes
- Local testing with Azurite emulator (Docker-based)
- Secure secret storage with per-user encryption

Files modified: 9 backend/frontend files
Files created: 5 test utility files
@netlify
Copy link

netlify bot commented Jan 24, 2026

Deploy Preview for flowfile-wasm canceled.

Name Link
🔨 Latest commit d49a0e9
🔍 Latest deploy log https://app.netlify.com/projects/flowfile-wasm/deploys/69750c5ee0d6d80008e58048

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants