Skip to content

add pipe and plugins isolation in table model #677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CritasWang merged 1 commit into from
Apr 1, 2025
Merged

add pipe and plugins isolation in table model #677

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ By declaratively configuring these three parts in an SQL statement, flexible dat
- Supports data synchronization from IoTDB version 1.x series to version 2.x and later.
- Does not support data synchronization from IoTDB version 2.x series to version 1.x series.
- When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends.
- The `pipe` and `pipe plugins` for tree models and table models are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support.

## 2. Usage Instructions

Expand Down
5 changes: 3 additions & 2 deletions src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ By declaratively configuring these three parts in an SQL statement, flexible dat
- Supports data synchronization from IoTDB version 1.x series to version 2.x and later.
- Does not support data synchronization from IoTDB version 2.x series to version 1.x series.
- When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends.
- The `pipe` and `pipe plugins` for tree models and table models are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support.

## 2. Usage Instructions

Expand Down Expand Up @@ -434,6 +435,7 @@ WITH SINK (
'node-urls' = '10.53.53.53:9780' -- URL of the DataNode service port on the target IoTDB
)
```

**Notes: Currently supported gateway models**
> For other models of network gateway devices, Please contact timechodb staff to confirm compatibility.

Expand All @@ -447,7 +449,7 @@ WITH SINK (
| Unknown | DPtech ISG5000 | No Limit | No Limit |
| Unknown | GAP‌‌
XL—GAP | No Limit | No Limit |

### 3.7 Compressed Synchronization

IoTDB supports specifying data compression methods during synchronization. The `compressor` parameter can be configured to enable real-time data compression and transmission. Supported algorithms include `snappy`, `gzip`, `lz4`, `zstd`, and `lzma2`. Multiple algorithms can be combined and applied in the configured order. The `rate-limit-bytes-per-second` parameter (supported in V1.3.3 and later) limits the maximum number of bytes transmitted per second (calculated after compression). If set to a value less than 0, there is no limit.
Expand All @@ -463,7 +465,6 @@ WITH SINK (
)
```


### 3.8 Encrypted Synchronization

IoTDB supports SSL encryption during synchronization to securely transmit data between IoTDB instances. By configuring SSL-related parameters such as the certificate path (`ssl.trust-store-path`) and password (`ssl.trust-store-pwd`), data can be protected by SSL encryption during synchronization.
Expand Down
1 change: 1 addition & 0 deletions src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ By declaratively configuring these three parts in an SQL statement, flexible dat
- Supports data synchronization from IoTDB version 1.x series to version 2.x and later.
- Does not support data synchronization from IoTDB version 2.x series to version 1.x series.
- When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends.
- The `pipe` and `pipe plugins` for tree models and table models are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support.

## 2. Usage Instructions

Expand Down
1 change: 1 addition & 0 deletions src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ By declaratively configuring these three parts in an SQL statement, flexible dat
- Supports data synchronization from IoTDB version 1.x series to version 2.x and later.
- Does not support data synchronization from IoTDB version 2.x series to version 1.x series.
- When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends.
- The `pipe` and `pipe plugins` for tree models and table models are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support.

## 2. Usage Instructions

Expand Down
2 changes: 1 addition & 1 deletion src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@
- 支持 1.x 系列版本 IoTDB 数据同步到 2.x 以及以上系列版本版本的 IoTDB。
- 不支持 2.x 系列版本 IoTDB 数据同步到 1.x 系列版本版本的 IoTDB。
- 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。
- 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。

## 2. 使用说明

Expand Down Expand Up @@ -554,4 +555,3 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - |
| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - |
| format | 数据传输的payload格式， 可选项包括：<br> - hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。<br> - tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。<br> - tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid |

1 change: 1 addition & 0 deletions src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@
- 支持 1.x 系列版本 IoTDB 数据同步到 2.x 以及以上系列版本版本的 IoTDB。
- 不支持 2.x 系列版本 IoTDB 数据同步到 1.x 系列版本版本的 IoTDB。
- 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。
- 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。

## 2. 使用说明

Expand Down
1 change: 1 addition & 0 deletions src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@
- 支持 1.x 系列版本 IoTDB 数据同步到 2.x 以及以上系列版本版本的 IoTDB。
- 不支持 2.x 系列版本 IoTDB 数据同步到 1.x 系列版本版本的 IoTDB。
- 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。
- 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。

## 2. 使用说明

Expand Down
1 change: 1 addition & 0 deletions src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@
- 支持 1.x 系列版本 IoTDB 数据同步到 2.x 以及以上系列版本版本的 IoTDB。
- 不支持 2.x 系列版本 IoTDB 数据同步到 1.x 系列版本版本的 IoTDB。
- 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。
- 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。

## 2. 使用说明

Expand Down