diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index df9ed3a38..e7f5fdc52 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -35,7 +35,47 @@ A data synchronization task consists of three stages: - Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement. - Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement. -By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. +By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. Currently, data synchronization supports the synchronization of the following information, and you can select the synchronization scope when creating a synchronization task (the default is data.insert, which means synchronizing newly written data): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Synchronization ScopeSynchronization Content Description
allAll scopes
data（Data）insertSynchronize newly written data
deleteSynchronize deleted data
schemadatabaseSynchronize database creation, modification or deletion operations
tableSynchronize table creation, modification or deletion operations
TTLSynchronize the data retention time
auth-Synchronize user permissions and access control
### 1.2 Functional Limitations and Notes @@ -468,17 +508,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter -| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | -| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- | -| source | iotdb-source | String: iotdb-source | Yes | - | -| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | -| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | -| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | -| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | -| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | -| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| forwarding-pipe-requests | Specifies whether to forward data that was synchronized via the pipe to external clusters. Typically used for setting up **active-active clusters**. In active-active cluster mode, this parameter should be set to `false` to prevent **infinite circular synchronization**. | Boolean: true / false | No | true | +| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | +| :----------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| :----------- | :---------------------------------------------------------- | +| source | iotdb-source | String: iotdb-source | Yes | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | - | +| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | +| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | +| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | +| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | +| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index 655087533..2529c5711 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -34,7 +34,47 @@ A data synchronization task consists of three stages: - Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement. - Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement. -By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. +By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved.Currently, data synchronization supports the synchronization of the following information, and you can select the synchronization scope when creating a synchronization task (the default is data.insert, which means synchronizing newly written data): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Synchronization ScopeSynchronization Content Description
allAll scopes
data（Data）insertSynchronize newly written data
deleteSynchronize deleted data
schemadatabaseSynchronize database creation, modification or deletion operations
tableSynchronize table creation, modification or deletion operations
TTLSynchronize the data retention time
auth-Synchronize user permissions and access control
### 1.2 Functional Limitations and Notes @@ -495,17 +535,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter -| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | -| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- | -| source | iotdb-source | String: iotdb-source | Yes | - | -| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | -| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | -| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | -| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | -| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | -| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| forwarding-pipe-requests | Specifies whether to forward data that was synchronized via the pipe to external clusters. Typically used for setting up **active-active clusters**. In active-active cluster mode, this parameter should be set to `false` to prevent **infinite circular synchronization**. | Boolean: true / false | No | true | +| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | +| :----------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| :----------- | :---------------------------------------------------------- | +| source | iotdb-source | String: iotdb-source | Yes | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data,schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | - | +| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | +| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | +| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | +| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | +| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index df9ed3a38..e7f5fdc52 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -35,7 +35,47 @@ A data synchronization task consists of three stages: - Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement. - Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement. -By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. +By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. Currently, data synchronization supports the synchronization of the following information, and you can select the synchronization scope when creating a synchronization task (the default is data.insert, which means synchronizing newly written data): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Synchronization ScopeSynchronization Content Description
allAll scopes
data（Data）insertSynchronize newly written data
deleteSynchronize deleted data
schemadatabaseSynchronize database creation, modification or deletion operations
tableSynchronize table creation, modification or deletion operations
TTLSynchronize the data retention time
auth-Synchronize user permissions and access control
### 1.2 Functional Limitations and Notes @@ -468,17 +508,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter -| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | -| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- | -| source | iotdb-source | String: iotdb-source | Yes | - | -| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | -| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | -| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | -| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | -| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | -| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| forwarding-pipe-requests | Specifies whether to forward data that was synchronized via the pipe to external clusters. Typically used for setting up **active-active clusters**. In active-active cluster mode, this parameter should be set to `false` to prevent **infinite circular synchronization**. | Boolean: true / false | No | true | +| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | +| :----------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| :----------- | :---------------------------------------------------------- | +| source | iotdb-source | String: iotdb-source | Yes | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | - | +| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | +| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | +| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | +| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | +| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index 655087533..2529c5711 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -34,7 +34,47 @@ A data synchronization task consists of three stages: - Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement. - Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement. -By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. +By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved.Currently, data synchronization supports the synchronization of the following information, and you can select the synchronization scope when creating a synchronization task (the default is data.insert, which means synchronizing newly written data): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Synchronization ScopeSynchronization Content Description
allAll scopes
data（Data）insertSynchronize newly written data
deleteSynchronize deleted data
schemadatabaseSynchronize database creation, modification or deletion operations
tableSynchronize table creation, modification or deletion operations
TTLSynchronize the data retention time
auth-Synchronize user permissions and access control
### 1.2 Functional Limitations and Notes @@ -495,17 +535,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter -| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | -| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- | -| source | iotdb-source | String: iotdb-source | Yes | - | -| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | -| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | -| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | -| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | -| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | -| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| forwarding-pipe-requests | Specifies whether to forward data that was synchronized via the pipe to external clusters. Typically used for setting up **active-active clusters**. In active-active cluster mode, this parameter should be set to `false` to prevent **infinite circular synchronization**. | Boolean: true / false | No | true | +| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | +| :----------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| :----------- | :---------------------------------------------------------- | +| source | iotdb-source | String: iotdb-source | Yes | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data,schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | - | +| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | +| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | +| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | +| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | +| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index 20e2a316b..d798beb00 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -34,7 +34,47 @@ - 处理（Process）阶段：该部分用于处理从源 IoTDB 抽取出的数据，在 SQL 语句中的 processor 部分定义 - 发送（Sink）阶段：该部分用于向目标 IoTDB 发送数据，在 SQL 语句中的 sink 部分定义 -通过 SQL 语句声明式地配置 3 个部分的具体内容，可实现灵活的数据同步能力。 +通过 SQL 语句声明式地配置 3 个部分的具体内容，可实现灵活的数据同步能力。目前数据同步支持以下信息的同步，您可以在创建同步任务时对同步范围进行选择（默认选择 data.insert，即同步新写入的数据）： + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
同步范围同步内容说明
all所有范围
data（数据）insert（增量）同步新写入的数据
delete（删除）同步被删除的数据
schema（元数据）database（数据库）同步数据库的创建、修改或删除操作
table（表）同步表的创建、修改或删除操作
TTL（数据到期时间）同步数据的存活时间
auth（权限）-同步用户权限和访问控制
### 1.2 功能限制及说明 @@ -460,17 +500,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### source 参数 -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -| ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------ | ------------------------------- | -| source | iotdb-source | String: iotdb-source | 必填 | - | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | -| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | -| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | -| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | -| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | -| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | -| forwarding-pipe-requests | 是否转发由 pipe 数据同步而来的集群外的数据。一般供搭建双活集群时使用，双活集群模式下该参数为 false，以此避免无限的环形同步。 | Boolean: true / false | 否 | true | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | +| source | iotdb-source | String: iotdb-source | 必填 | - | +| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | data.insert | +| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | +| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | +| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | +| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | +| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | +| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | +| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | +| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| mode.double-living | 是否开启全量双活模式，开启后将忽略`-sql_dialect`连接方式，树表模型数据均会被捕获，且不会转发由另一pipe同步而来的数据。 | Boolean: true / false | 否 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index fc65c8d9f..943716dc9 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -34,7 +34,47 @@ - 处理（Process）阶段：该部分用于处理从源 IoTDB 抽取出的数据，在 SQL 语句中的 processor 部分定义 - 发送（Sink）阶段：该部分用于向目标 IoTDB 发送数据，在 SQL 语句中的 sink 部分定义 -通过 SQL 语句声明式地配置 3 个部分的具体内容，可实现灵活的数据同步能力。 +通过 SQL 语句声明式地配置 3 个部分的具体内容，可实现灵活的数据同步能力。目前数据同步支持以下信息的同步，您可以在创建同步任务时对同步范围进行选择（默认选择 data.insert，即同步新写入的数据）： + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
同步范围同步内容说明
all所有范围
data（数据）insert（增量）同步新写入的数据
delete（删除）同步被删除的数据
schema（元数据）database（数据库）同步数据库的创建、修改或删除操作
table（表）同步表的创建、修改或删除操作
TTL（数据到期时间）同步数据的存活时间
auth（权限）-同步用户权限和访问控制
### 1.2 功能限制及说明 @@ -502,17 +542,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### source 参数 -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -| ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------ | ------------------------------- | -| source | iotdb-source | String: iotdb-source | 必填 | - | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | -| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | -| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | -| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | -| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | -| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | -| forwarding-pipe-requests | 是否转发由 pipe 数据同步而来的集群外的数据。一般供搭建双活集群时使用，双活集群模式下该参数为 false，以此避免无限的环形同步。 | Boolean: true / false | 否 | true | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | +| source | iotdb-source | String: iotdb-source | 必填 | - | +| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl),auth | 选填 | data.insert | +| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | +| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | +| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | +| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | +| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | +| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | +| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | +| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| mode.double-living | 是否开启全量双活模式，开启后将忽略`-sql_dialect`连接方式，树表模型数据均会被捕获，且不会转发由另一pipe同步而来的数据。 | Boolean: true / false | 否 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index 20e2a316b..d798beb00 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -34,7 +34,47 @@ - 处理（Process）阶段：该部分用于处理从源 IoTDB 抽取出的数据，在 SQL 语句中的 processor 部分定义 - 发送（Sink）阶段：该部分用于向目标 IoTDB 发送数据，在 SQL 语句中的 sink 部分定义 -通过 SQL 语句声明式地配置 3 个部分的具体内容，可实现灵活的数据同步能力。 +通过 SQL 语句声明式地配置 3 个部分的具体内容，可实现灵活的数据同步能力。目前数据同步支持以下信息的同步，您可以在创建同步任务时对同步范围进行选择（默认选择 data.insert，即同步新写入的数据）： + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
同步范围同步内容说明
all所有范围
data（数据）insert（增量）同步新写入的数据
delete（删除）同步被删除的数据
schema（元数据）database（数据库）同步数据库的创建、修改或删除操作
table（表）同步表的创建、修改或删除操作
TTL（数据到期时间）同步数据的存活时间
auth（权限）-同步用户权限和访问控制
### 1.2 功能限制及说明 @@ -460,17 +500,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### source 参数 -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -| ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------ | ------------------------------- | -| source | iotdb-source | String: iotdb-source | 必填 | - | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | -| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | -| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | -| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | -| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | -| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | -| forwarding-pipe-requests | 是否转发由 pipe 数据同步而来的集群外的数据。一般供搭建双活集群时使用，双活集群模式下该参数为 false，以此避免无限的环形同步。 | Boolean: true / false | 否 | true | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | +| source | iotdb-source | String: iotdb-source | 必填 | - | +| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | data.insert | +| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | +| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | +| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | +| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | +| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | +| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | +| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | +| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| mode.double-living | 是否开启全量双活模式，开启后将忽略`-sql_dialect`连接方式，树表模型数据均会被捕获，且不会转发由另一pipe同步而来的数据。 | Boolean: true / false | 否 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index fc65c8d9f..943716dc9 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -34,7 +34,47 @@ - 处理（Process）阶段：该部分用于处理从源 IoTDB 抽取出的数据，在 SQL 语句中的 processor 部分定义 - 发送（Sink）阶段：该部分用于向目标 IoTDB 发送数据，在 SQL 语句中的 sink 部分定义 -通过 SQL 语句声明式地配置 3 个部分的具体内容，可实现灵活的数据同步能力。 +通过 SQL 语句声明式地配置 3 个部分的具体内容，可实现灵活的数据同步能力。目前数据同步支持以下信息的同步，您可以在创建同步任务时对同步范围进行选择（默认选择 data.insert，即同步新写入的数据）： + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
同步范围同步内容说明
all所有范围
data（数据）insert（增量）同步新写入的数据
delete（删除）同步被删除的数据
schema（元数据）database（数据库）同步数据库的创建、修改或删除操作
table（表）同步表的创建、修改或删除操作
TTL（数据到期时间）同步数据的存活时间
auth（权限）-同步用户权限和访问控制
### 1.2 功能限制及说明 @@ -502,17 +542,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### source 参数 -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -| ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------ | ------------------------------- | -| source | iotdb-source | String: iotdb-source | 必填 | - | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | -| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | -| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | -| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | -| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | -| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | -| forwarding-pipe-requests | 是否转发由 pipe 数据同步而来的集群外的数据。一般供搭建双活集群时使用，双活集群模式下该参数为 false，以此避免无限的环形同步。 | Boolean: true / false | 否 | true | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | +| source | iotdb-source | String: iotdb-source | 必填 | - | +| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl),auth | 选填 | data.insert | +| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | +| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | +| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | +| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | +| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | +| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | +| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | +| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| mode.double-living | 是否开启全量双活模式，开启后将忽略`-sql_dialect`连接方式，树表模型数据均会被捕获，且不会转发由另一pipe同步而来的数据。 | Boolean: true / false | 否 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐