Skip to content

Multi level storage adapted to non AWS object storage systems complia… #676

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CritasWang merged 1 commit into from
Mar 28, 2025
Merged

Multi level storage adapted to non AWS object storage systems complia… #676

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions src/UserGuide/Master/Table/User-Manual/Tiered-Storage_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,13 @@ The specific parameter definitions and their descriptions are as follows.
| `dn_data_dirs` | `data/datanode/data` | Specifies storage directories grouped into tiers. | Tiers are separated by `;`, directories within the same tier are separated by `,`. <br>Cloud storage (e.g., AWS S3) can only be the last tier. <br>Use `OBJECT_STORAGE` to denote cloud storage. <br>Only one cloud storage bucket is allowed. |
| `tier_ttl_in_ms` | `-1` | Defines the TTL (in milliseconds) for each tier to determine the data range it manages. | Tiers are separated by `;`. <br>The number of tiers must match `dn_data_dirs`. <br>`-1` means "no limit". |
| `dn_default_space_usage_thresholds` | `0.85` | Defines the minimum remaining space threshold (as a ratio) for each tier. When a tier’s remaining space falls below this threshold, data is migrated to the next tier. <br>The last tier triggers `READ_ONLY` mode. | -Tiers are separated by `;`.The number of tiers must match `dn_data_dirs`. |
| `object_storage_type` | `AWS_S3` | Cloud storage type. | Only `AWS_S3` is supported. |
| `object_storage_type` | `AWS_S3` | Cloud storage type. | all `AWS_S3` is supported. |
| `object_storage_bucket` | `iotdb_data` | Cloud storage bucket name. | Required only if cloud storage is used. |
| `object_storage_endpoiont` | (Empty) | Cloud storage endpoint. | Required only if cloud storage is used. |
| `object_storage_region` | (Empty) | Cloud storage Region. | Required only if cloud storage is used. |
| `object_storage_access_key` | (Empty) | Cloud storage access key. | Required only if cloud storage is used. |
| `object_storage_access_secret` | (Empty) | Cloud storage access secret. | Required only if cloud storage is used. |
| `enable_path_style_access` | false | Whether to enable path style access for object storage service. | Required only if cloud storage is used. |
| `remote_tsfile_cache_dirs` | `data/datanode/data/cache` | Local cache directory for cloud storage. | Required only if cloud storage is used. |
| `remote_tsfile_cache_page_size_in_kb` | `20480` | Page size (in KB) for cloud storage local cache. | Required only if cloud storage is used. |
| `remote_tsfile_cache_max_disk_usage_in_mb` | `51200` | Maximum disk space (in MB) allocated for cloud storage local cache. | Required only if cloud storage is used. |
Expand Down Expand Up @@ -79,11 +81,13 @@ tier_ttl_in_ms=86400000;864000000;-1
dn_default_space_usage_thresholds=0.2;0.15;0.1
object_storage_type=AWS_S3
object_storage_bucket=iotdb
object_storage_region=<your_region>
object_storage_endpoiont=<your_endpoint>
object_storage_access_key=<your_access_key>
object_storage_access_secret=<your_access_secret>

# Optional configurations
enable_path_style_access=false
remote_tsfile_cache_dirs=data/datanode/data/cache
remote_tsfile_cache_page_size_in_kb=20971520
remote_tsfile_cache_max_disk_usage_in_mb=53687091200
Expand All @@ -95,4 +99,4 @@ remote_tsfile_cache_max_disk_usage_in_mb=53687091200
| :------- | :--------------------------- | :----------------------------- | :---------------------------- |
| Tier 1 | `/data1/data` | Last 1 day of data | 20% |
| Tier 2 | `/data2/data`, `/data3/data` | Data from 1 day to 10 days ago | 15% |
| Tier 3 | AWS S3 Cloud Storage | Data older than 10 days | 10% |
| Tier 3 | S3 Cloud Storage | Data older than 10 days | 10% |
10 changes: 7 additions & 3 deletions src/UserGuide/Master/Tree/User-Manual/Tiered-Storage_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,13 @@ The specific parameter definitions and their descriptions are as follows.
| dn_data_dirs | data/datanode/data | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud (OBJECT_STORAGE) configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE |
| tier_ttl_in_ms | -1 | Define the maximum age of data for which each tier is responsible | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs；"-1" means "unlimited". |
| dn_default_space_usage_thresholds | 0.85 | Define the minimum remaining space ratio for each tier data catalogue; when the remaining space is less than this ratio, the data will be automatically migrated to the next tier; when the remaining storage space of the last tier falls below this threshold, the system will be set to READ_ONLY | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
| object_storage_type | AWS_S3 | Cloud Storage Type | IoTDB currently only supports AWS S3 as a remote storage type, and this parameter can't be modified |
| object_storage_type | `AWS_S3` | Cloud storage type. | all `AWS_S3` is supported. |
| object_storage_bucket | iotdb_data | Name of cloud storage bucket | Bucket definition in AWS S3; no need to configure if remote storage is not used |
| object_storage_endpoiont | | endpoint of cloud storage | endpoint of AWS S3；If remote storage is not used, no configuration required |
| object_storage_region | (Empty) | Cloud storage Region. | Required only if cloud storage is used. |
| object_storage_access_key | | Authentication information stored in the cloud: key | AWS S3 credential key；If remote storage is not used, no configuration required |
| object_storage_access_secret | | Authentication information stored in the cloud: secret | AWS S3 credential secret；If remote storage is not used, no configuration required |
| enable_path_style_access | false | Whether to enable path style access for object storage service. | Required only if cloud storage is used. |
| remote_tsfile_cache_dirs | data/datanode/data/cache | Cache directory stored locally in the cloud | If remote storage is not used, no configuration required |
| remote_tsfile_cache_page_size_in_kb | 20480 |Block size of locally cached files stored in the cloud | If remote storage is not used, no configuration required |
| remote_tsfile_cache_max_disk_usage_in_mb | 51200 | Maximum Disk Occupancy Size for Cloud Storage Local Cache | If remote storage is not used, no configuration required |
Expand Down Expand Up @@ -75,13 +77,15 @@ The following takes three-level storage as an example:
dn_data_dirs=/data1/data;/data2/data,/data3/data;OBJECT_STORAGE
tier_ttl_in_ms=86400000;864000000;-1
dn_default_space_usage_thresholds=0.2;0.15;0.1
object_storage_name=AWS_S3
object_storage_type=AWS_S3
object_storage_bucket=iotdb
object_storage_region=<your_region>
object_storage_endpoiont=<your_endpoint>
object_storage_access_key=<your_access_key>
object_storage_access_secret=<your_access_secret>

// Optional configuration items
enable_path_style_access=false
remote_tsfile_cache_dirs=data/datanode/data/cache
remote_tsfile_cache_page_size_in_kb=20971520
remote_tsfile_cache_max_disk_usage_in_mb=53687091200
Expand All @@ -93,4 +97,4 @@ In this example, a total of three levels of storage are configured, specifically
| -------- | -------------------------------------- | ---------------------------- | ------------------------ |
| tier1 | path 1：/data1/data | data for last 1 day | 20% |
| tier2 | path 1：/data2/data path 2：/data3/data | data from past 1 day to past 10 days | 15% |
| tier3 | Remote AWS S3 Storage | data from 10 days ago | 10% |
| tier 3 | S3 Cloud Storage | Data older than 10 days | 10% |
Loading