Skip to content

add schema sync in table model #670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
leto-b wants to merge 1 commit into from
Closed

add schema sync in table model #670

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,43 @@ A data synchronization task consists of three stages:
- Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement.
- Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement.

By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved.
By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. Currently, data synchronization supports the synchronization of the following information, and you can select the synchronization scope when creating a synchronization task (the default is data.insert, which means synchronizing newly written data):

<table style="text-align: left;">
<tbody>
<tr> <th>Synchronization Scope</th>
<th>Synchronization Content </th>
<th>Description</th>
</tr>
<tr>
<td colspan="2">all</td>
<td>All scopes</td>
</tr>
<tr>
<td rowspan="1">data（Data）</td>
<td>insert</td>
<td>Synchronize newly written data</td>
</tr>
<tr>
<td rowspan="3">schema</td>
<td>database</td>
<td>Synchronize database creation, modification or deletion operations</td>
</tr>
<tr>
<td>table</td>
<td>Synchronize table creation, modification or deletion operations</td>
</tr>
<tr>
<td>TTL</td>
<td>Synchronize the data retention time</td>
</tr>
<tr>
<td>auth</td>
<td>-</td>
<td>Synchronize user permissions and access control</td>
</tr>
</tbody>
</table>

### 1.2 Functional Limitations and Notes

Expand Down Expand Up @@ -471,6 +507,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** |
| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- |
| source | iotdb-source | String: iotdb-source | Yes | - |
| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert), schema(database,timeseries,ttl), auth | Optional | data.insert |
| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert), schema(database,timeseries,ttl), auth | Optional | - |
| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true |
| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true |
| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false |
Expand Down
40 changes: 39 additions & 1 deletion src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,43 @@ A data synchronization task consists of three stages:
- Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement.
- Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement.

By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved.
By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. Currently, data synchronization supports the synchronization of the following information, and you can select the synchronization scope when creating a synchronization task (the default is data.insert, which means synchronizing newly written data):

<table style="text-align: left;">
<tbody>
<tr> <th>Synchronization Scope</th>
<th>Synchronization Content </th>
<th>Description</th>
</tr>
<tr>
<td colspan="2">all</td>
<td>All scopes</td>
</tr>
<tr>
<td rowspan="1">data（Data）</td>
<td>insert</td>
<td>Synchronize newly written data</td>
</tr>
<tr>
<td rowspan="3">schema</td>
<td>database</td>
<td>Synchronize database creation, modification or deletion operations</td>
</tr>
<tr>
<td>table</td>
<td>Synchronize table creation, modification or deletion operations</td>
</tr>
<tr>
<td>TTL</td>
<td>Synchronize the data retention time</td>
</tr>
<tr>
<td>auth</td>
<td>-</td>
<td>Synchronize user permissions and access control</td>
</tr>
</tbody>
</table>

### 1.2 Functional Limitations and Notes

Expand Down Expand Up @@ -498,6 +534,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** |
| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- |
| source | iotdb-source | String: iotdb-source | Yes | - |
| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert), schema(database,timeseries,ttl), auth | Optional | data.insert |
| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert), schema(database,timeseries,ttl), auth | Optional | - |
| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true |
| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true |
| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false |
Expand Down
40 changes: 39 additions & 1 deletion src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,43 @@ A data synchronization task consists of three stages:
- Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement.
- Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement.

By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved.
By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. Currently, data synchronization supports the synchronization of the following information, and you can select the synchronization scope when creating a synchronization task (the default is data.insert, which means synchronizing newly written data):

<table style="text-align: left;">
<tbody>
<tr> <th>Synchronization Scope</th>
<th>Synchronization Content </th>
<th>Description</th>
</tr>
<tr>
<td colspan="2">all</td>
<td>All scopes</td>
</tr>
<tr>
<td rowspan="1">data（Data）</td>
<td>insert</td>
<td>Synchronize newly written data</td>
</tr>
<tr>
<td rowspan="3">schema</td>
<td>database</td>
<td>Synchronize database creation, modification or deletion operations</td>
</tr>
<tr>
<td>table</td>
<td>Synchronize table creation, modification or deletion operations</td>
</tr>
<tr>
<td>TTL</td>
<td>Synchronize the data retention time</td>
</tr>
<tr>
<td>auth</td>
<td>-</td>
<td>Synchronize user permissions and access control</td>
</tr>
</tbody>
</table>

### 1.2 Functional Limitations and Notes

Expand Down Expand Up @@ -471,6 +507,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** |
| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- |
| source | iotdb-source | String: iotdb-source | Yes | - |
| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert), schema(database,timeseries,ttl), auth | Optional | data.insert |
| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert), schema(database,timeseries,ttl), auth | Optional | - |
| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true |
| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true |
| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false |
Expand Down
Loading