Skip to content

[docs] Document Fluss S3 writing mechanism#1

Draft
Copilot wants to merge 1 commit intomainfrom
copilot/explore-fluss-s3-integration
Draft

[docs] Document Fluss S3 writing mechanism#1
Copilot wants to merge 1 commit intomainfrom
copilot/explore-fluss-s3-integration

Conversation

Copy link

Copilot AI commented Nov 28, 2025

Purpose

Linked issue: N/A (research/documentation task)

This PR documents how Fluss writes data to S3, analyzing the architecture and data flow.

Brief change log

Research findings on Fluss S3 integration:

Architecture Layers:

  • FileSystemHadoopFileSystemS3FileSystem → Hadoop S3AFileSystem → AWS S3

Key Components:

  • S3FileSystemPlugin / S3AFileSystemPlugin: Handle s3:// and s3a:// URI schemes
  • DefaultRemoteLogStorage: Orchestrates log segment copying to remote storage
  • FsRemoteLogOutputStream: Buffered write stream to S3
  • LogTieringTask: Background task for log tiering

Data Flow:

Local Log Segment → LogTieringTask → DefaultRemoteLogStorage → FsRemoteLogOutputStream → S3

Remote Directory Structure:

{remote.data.dir}/log/{db}/{table}_{tableId}/{bucketId}/{segment_uuid}/
    ├── {offset}.log
    ├── {offset}.index
    ├── {offset}.timeindex
    └── {offset}.writer_snapshot

Authentication: Supports static credentials (fs.s3a.access.key) or dynamic STS session credentials via S3DelegationTokenProvider.

Tests

N/A - research/documentation only

API and Format

No changes

Documentation

No code changes - this PR documents existing S3 integration for reference.

Original prompt

查看fluss中是怎么写入s3的


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI changed the title [WIP] Investigate Fluss write operations to S3 [docs] Document Fluss S3 writing mechanism Nov 28, 2025
Copilot AI requested a review from Denovo1998 November 28, 2025 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants