From 442af3a3db97b1bee250fd9cb2c486f41e4027f9 Mon Sep 17 00:00:00 2001 From: W1y1r <2730956796@qq.com> Date: Thu, 20 Feb 2025 18:37:31 +0800 Subject: [PATCH 1/4] Quickly get started with English document PR --- .../Table/QuickStart/QuickStart_apache.md | 78 +++++++++++++++++++ .../Table/QuickStart/QuickStart_timecho.md | 74 +++++++++++++++++- .../QuickStart/QuickStart_apache.md | 58 +++++++++++++- .../QuickStart/QuickStart_timecho.md | 74 +++++++++++++++++- .../Table/QuickStart/QuickStart_timecho.md | 2 +- .../QuickStart/QuickStart_timecho.md | 2 +- 6 files changed, 283 insertions(+), 5 deletions(-) create mode 100644 src/UserGuide/Master/Table/QuickStart/QuickStart_apache.md diff --git a/src/UserGuide/Master/Table/QuickStart/QuickStart_apache.md b/src/UserGuide/Master/Table/QuickStart/QuickStart_apache.md new file mode 100644 index 000000000..2df10e79f --- /dev/null +++ b/src/UserGuide/Master/Table/QuickStart/QuickStart_apache.md @@ -0,0 +1,78 @@ + + +# Quick Start + +This document will guide you through methods to get started quickly with IoTDB. + +## How to Install and Deploy? + +This guide will assist you in quickly installing and deploying IoTDB. You can quickly navigate to the content you need to review through the following document links: + +1. Prepare the necessary machine resources: The deployment and operation of IoTDB require consideration of various aspects of machine resource configuration. For specific resource configurations, please refer to [Database Resource](../Deployment-and-Maintenance/Database-Resources.md) + +2. Complete system configuration preparations: IoTDB's system configuration involves multiple aspects. For an introduction to key system configurations, please see [System Requirements](../Deployment-and-Maintenance/Environment-Requirements.md) + +3. Obtain the installation package: You can obtain the IoTDB installation package on the [Apache IoTDB official website](https://iotdb.apache.org/zh/Download/).For the specific structure of the installation package, please refer to[Obtain TimechoDB](../Deployment-and-Maintenance/IoTDB-Package_apache.md) + +4. Install the database and activate it: Depending on your actual deployment architecture, you can choose from the following tutorials for installation and deployment: + + - Stand-Alone Deployment: [Stand-Alone Deployment ](../Deployment-and-Maintenance/Stand-Alone-Deployment_apache.md) + + - Cluster Deployment:[Cluster Deployment](../Deployment-and-Maintenance/Cluster-Deployment_apache.md) + +> ❗️Note: We currently still recommend direct installation and deployment on physical/virtual machines. For Docker deployment, please refer to [Docker Deployment](../Deployment-and-Maintenance/Docker-Deployment_apache.md) + +## How to Use IoTDB? + +1. Database Modeling Design: Database modeling is a crucial step in creating a database system, involving the design of data structures and relationships to ensure that the organization of data meets the needs of specific applications. The following documents will help you quickly understand IoTDB's modeling design: + + - Introduction to Time Series Concepts: [Navigating Time Series Data](../Background-knowledge/Navigating_Time_Series_Data.md) + + - Introduction to Modeling Design:[Data Model and Terminology](../Background-knowledge/Data-Model-and-Terminology_apache.md) + + - Introduction to Database:[Database Management](../Basic-Concept/Database-Management.md) + + - Introduction to Tables: [Table Management](../Basic-Concept/Table-Management.md) + +2. Data Insertion & Updates: IoTDB provides multiple methods for inserting real-time data. For basic data insertion and updating operations, please see [Write&Updata Data](../Basic-Concept/Write-Updata-Data.md) + +3. Data Querying: IoTDB offers a rich set of data querying capabilities. For a basic introduction to data querying, please see [Query Data](../Basic-Concept/Query-Data.md) + +4. Data Deletion: IoTDB supports two deletion methods: SQL-based deletion and automatic expiration deletion (TTL). + + - SQL-Based Deletion: For a basic introduction, please refer to [Delete Data](../Basic-Concept/Delete-Data.md) + + - Automatic Expiration Deletion (TTL): For a basic introduction, please see [TTL Delete Data](../Basic-Concept/TTL-Delete-Data.md) + +5. Advanced Features: In addition to common database functions such as insertion and querying, IoTDB also supports features like "data synchronization." For specific usage methods, please refer to the respective documents: + + - Data Synchronization: [Data Sync](../User-Manual/Data-Sync_apache.md) + +6. Application Programming Interfaces (APIs): IoTDB provides various application programming interfaces (APIs) to facilitate developers' interaction with IoTDB in applications. Currently supported interfaces include [Java Native API](../API/Programming-Java-Native-API.md)、[Python Native API](../API/Programming-Python-Native-API.md)、[JDBC](../API/Programming-JDBC.md), and more. For more programming interfaces, please refer to the [Application Programming Interfaces] section on the official website. + +## Want to learn more technical details? + +If you want to explore IoTDB’s internal mechanisms further, refer to the following documents: + + - Data Partitioning and Load Balancing: IoTDB is designed with a partitioning strategy and load balancing algorithm to enhance cluster availability and performance. For more details, please see [Cluster data partitioning](../Technical-Insider/Cluster-data-partitioning.md) + + - Compression & Encoding: IoTDB employs various encoding and compression techniques optimized for different data types to improve storage efficiency. For more details, please see [Encoding and Compression](../Technical-Insider/Encoding-and-Compression.md) diff --git a/src/UserGuide/Master/Table/QuickStart/QuickStart_timecho.md b/src/UserGuide/Master/Table/QuickStart/QuickStart_timecho.md index 3ac1654f7..0c47420c0 100644 --- a/src/UserGuide/Master/Table/QuickStart/QuickStart_timecho.md +++ b/src/UserGuide/Master/Table/QuickStart/QuickStart_timecho.md @@ -19,4 +19,76 @@ --> -**Comming Soon** \ No newline at end of file +# Quick Start + +This document will guide you through methods to get started quickly with IoTDB. + +## How to Install and Deploy? + +This guide will assist you in quickly installing and deploying IoTDB. You can quickly navigate to the content you need to review through the following document links: + +1. Prepare the necessary machine resources: The deployment and operation of IoTDB require consideration of various aspects of machine resource configuration. For specific resource configurations, please refer to [Database Resource](../Deployment-and-Maintenance/Database-Resources.md) + +2. Complete system configuration preparations: IoTDB's system configuration involves multiple aspects. For an introduction to key system configurations, please see [System Requirements](../Deployment-and-Maintenance/Environment-Requirements.md) + +3. Obtain the installation package: You can contact the Timecho Team to get the IoTDB installation package to ensure you download the latest and most stable version. For the specific structure of the installation package, please refer to[Obtain TimechoDB](../Deployment-and-Maintenance/IoTDB-Package_timecho.md) + +4. Install the database and activate it: Depending on your actual deployment architecture, you can choose from the following tutorials for installation and deployment: + + - Stand-Alone Deployment: [Stand-Alone Deployment ](../Deployment-and-Maintenance/Stand-Alone-Deployment_timecho.md) + + - Cluster Deployment:[Cluster Deployment](../Deployment-and-Maintenance/Cluster-Deployment_timecho.md) + + - Dual-Active Deployment:[Dual-Active Deployment](../Deployment-and-Maintenance/Dual-Active-Deployment_timecho.md) + +> ❗️Note: We currently still recommend direct installation and deployment on physical/virtual machines. For Docker deployment, please refer to [Docker Deployment](../Deployment-and-Maintenance/Docker-Deployment_timecho.md) + +5. Install supporting tools: The TimechoDB provides supporting tools such as a monitoring panel, which is recommended to be installed when deploying the TimechoDB to facilitate a more convenient usage: + + - Monitoring Panel: Provides hundreds of database monitoring metrics for fine-grained monitoring of IoTDB and its operating system, assisting in system optimization, performance tuning, and bottleneck detection. For installation steps, please see [Monitoring-panel Deployment ](../Deployment-and-Maintenance/Monitoring-panel-deployment.md) + + +## How to Use IoTDB? + +1. Database Modeling Design: Database modeling is a crucial step in creating a database system, involving the design of data structures and relationships to ensure that the organization of data meets the needs of specific applications. The following documents will help you quickly understand IoTDB's modeling design: + + - Introduction to Time Series Concepts: [Navigating Time Series Data](../Background-knowledge/Navigating_Time_Series_Data.md) + + - Introduction to Modeling Design:[Data Model and Terminology](../Background-knowledge/Data-Model-and-Terminology_timecho.md) + + - Introduction to Database:[Database Management](../Basic-Concept/Database-Management.md) + + - Introduction to Tables: [Table Management](../Basic-Concept/Table-Management.md) + +2. Data Insertion & Updates: IoTDB provides multiple methods for inserting real-time data. For basic data insertion and updating operations, please see [Write&Updata Data](../Basic-Concept/Write-Updata-Data.md) + +3. Data Querying: IoTDB offers a rich set of data querying capabilities. For a basic introduction to data querying, please see [Query Data](../Basic-Concept/Query-Data.md) + +4. Data Deletion: IoTDB supports two deletion methods: SQL-based deletion and automatic expiration deletion (TTL). + + - SQL-Based Deletion: For a basic introduction, please refer to [Delete Data](../Basic-Concept/Delete-Data.md) + + - Automatic Expiration Deletion (TTL): For a basic introduction, please see [TTL Delete Data](../Basic-Concept/TTL-Delete-Data.md) + +5. Advanced Features: In addition to common database functions such as insertion and querying, IoTDB also supports features like "data synchronization." For specific usage methods, please refer to the respective documents: + + - Data Synchronization: [Data Sync](../User-Manual/Data-Sync_timecho.md) + +6. Application Programming Interfaces (APIs): IoTDB provides various application programming interfaces (APIs) to facilitate developers' interaction with IoTDB in applications. Currently supported interfaces include [Java Native API](../API/Programming-Java-Native-API.md)、[Python Native API](../API/Programming-Python-Native-API.md)、[JDBC](../API/Programming-JDBC.md), and more. For more programming interfaces, please refer to the [Application Programming Interfaces] section on the official website. + +## What other convenient peripheral tools are available? + +In addition to its own rich features, IoTDB is complemented by a comprehensive suite of peripheral tools. This guide will help you quickly understand and use these tools: + + - Monitoring Panel: A tool for detailed monitoring of IoTDB and its operating system, covering hundreds of database monitoring indicators such as database performance and system resources, helping system optimization and bottleneck identification. For a detailed usage introduction, please refer to [Monitor Tool](../Tools-System/Monitor-Tool_timecho.md) + + +## Want to learn more technical details? + +If you want to explore IoTDB’s internal mechanisms further, refer to the following documents: + + - Data Partitioning and Load Balancing: IoTDB is designed with a partitioning strategy and load balancing algorithm to enhance cluster availability and performance. For more details, please see [Cluster data partitioning](../Technical-Insider/Cluster-data-partitioning.md) + + - Compression & Encoding: IoTDB employs various encoding and compression techniques optimized for different data types to improve storage efficiency. For more details, please see [Encoding and Compression](../Technical-Insider/Encoding-and-Compression.md) + + diff --git a/src/UserGuide/latest-Table/QuickStart/QuickStart_apache.md b/src/UserGuide/latest-Table/QuickStart/QuickStart_apache.md index 6f09a2609..2df10e79f 100644 --- a/src/UserGuide/latest-Table/QuickStart/QuickStart_apache.md +++ b/src/UserGuide/latest-Table/QuickStart/QuickStart_apache.md @@ -19,4 +19,60 @@ --> -# Comming Soon \ No newline at end of file +# Quick Start + +This document will guide you through methods to get started quickly with IoTDB. + +## How to Install and Deploy? + +This guide will assist you in quickly installing and deploying IoTDB. You can quickly navigate to the content you need to review through the following document links: + +1. Prepare the necessary machine resources: The deployment and operation of IoTDB require consideration of various aspects of machine resource configuration. For specific resource configurations, please refer to [Database Resource](../Deployment-and-Maintenance/Database-Resources.md) + +2. Complete system configuration preparations: IoTDB's system configuration involves multiple aspects. For an introduction to key system configurations, please see [System Requirements](../Deployment-and-Maintenance/Environment-Requirements.md) + +3. Obtain the installation package: You can obtain the IoTDB installation package on the [Apache IoTDB official website](https://iotdb.apache.org/zh/Download/).For the specific structure of the installation package, please refer to[Obtain TimechoDB](../Deployment-and-Maintenance/IoTDB-Package_apache.md) + +4. Install the database and activate it: Depending on your actual deployment architecture, you can choose from the following tutorials for installation and deployment: + + - Stand-Alone Deployment: [Stand-Alone Deployment ](../Deployment-and-Maintenance/Stand-Alone-Deployment_apache.md) + + - Cluster Deployment:[Cluster Deployment](../Deployment-and-Maintenance/Cluster-Deployment_apache.md) + +> ❗️Note: We currently still recommend direct installation and deployment on physical/virtual machines. For Docker deployment, please refer to [Docker Deployment](../Deployment-and-Maintenance/Docker-Deployment_apache.md) + +## How to Use IoTDB? + +1. Database Modeling Design: Database modeling is a crucial step in creating a database system, involving the design of data structures and relationships to ensure that the organization of data meets the needs of specific applications. The following documents will help you quickly understand IoTDB's modeling design: + + - Introduction to Time Series Concepts: [Navigating Time Series Data](../Background-knowledge/Navigating_Time_Series_Data.md) + + - Introduction to Modeling Design:[Data Model and Terminology](../Background-knowledge/Data-Model-and-Terminology_apache.md) + + - Introduction to Database:[Database Management](../Basic-Concept/Database-Management.md) + + - Introduction to Tables: [Table Management](../Basic-Concept/Table-Management.md) + +2. Data Insertion & Updates: IoTDB provides multiple methods for inserting real-time data. For basic data insertion and updating operations, please see [Write&Updata Data](../Basic-Concept/Write-Updata-Data.md) + +3. Data Querying: IoTDB offers a rich set of data querying capabilities. For a basic introduction to data querying, please see [Query Data](../Basic-Concept/Query-Data.md) + +4. Data Deletion: IoTDB supports two deletion methods: SQL-based deletion and automatic expiration deletion (TTL). + + - SQL-Based Deletion: For a basic introduction, please refer to [Delete Data](../Basic-Concept/Delete-Data.md) + + - Automatic Expiration Deletion (TTL): For a basic introduction, please see [TTL Delete Data](../Basic-Concept/TTL-Delete-Data.md) + +5. Advanced Features: In addition to common database functions such as insertion and querying, IoTDB also supports features like "data synchronization." For specific usage methods, please refer to the respective documents: + + - Data Synchronization: [Data Sync](../User-Manual/Data-Sync_apache.md) + +6. Application Programming Interfaces (APIs): IoTDB provides various application programming interfaces (APIs) to facilitate developers' interaction with IoTDB in applications. Currently supported interfaces include [Java Native API](../API/Programming-Java-Native-API.md)、[Python Native API](../API/Programming-Python-Native-API.md)、[JDBC](../API/Programming-JDBC.md), and more. For more programming interfaces, please refer to the [Application Programming Interfaces] section on the official website. + +## Want to learn more technical details? + +If you want to explore IoTDB’s internal mechanisms further, refer to the following documents: + + - Data Partitioning and Load Balancing: IoTDB is designed with a partitioning strategy and load balancing algorithm to enhance cluster availability and performance. For more details, please see [Cluster data partitioning](../Technical-Insider/Cluster-data-partitioning.md) + + - Compression & Encoding: IoTDB employs various encoding and compression techniques optimized for different data types to improve storage efficiency. For more details, please see [Encoding and Compression](../Technical-Insider/Encoding-and-Compression.md) diff --git a/src/UserGuide/latest-Table/QuickStart/QuickStart_timecho.md b/src/UserGuide/latest-Table/QuickStart/QuickStart_timecho.md index 6f09a2609..0c47420c0 100644 --- a/src/UserGuide/latest-Table/QuickStart/QuickStart_timecho.md +++ b/src/UserGuide/latest-Table/QuickStart/QuickStart_timecho.md @@ -19,4 +19,76 @@ --> -# Comming Soon \ No newline at end of file +# Quick Start + +This document will guide you through methods to get started quickly with IoTDB. + +## How to Install and Deploy? + +This guide will assist you in quickly installing and deploying IoTDB. You can quickly navigate to the content you need to review through the following document links: + +1. Prepare the necessary machine resources: The deployment and operation of IoTDB require consideration of various aspects of machine resource configuration. For specific resource configurations, please refer to [Database Resource](../Deployment-and-Maintenance/Database-Resources.md) + +2. Complete system configuration preparations: IoTDB's system configuration involves multiple aspects. For an introduction to key system configurations, please see [System Requirements](../Deployment-and-Maintenance/Environment-Requirements.md) + +3. Obtain the installation package: You can contact the Timecho Team to get the IoTDB installation package to ensure you download the latest and most stable version. For the specific structure of the installation package, please refer to[Obtain TimechoDB](../Deployment-and-Maintenance/IoTDB-Package_timecho.md) + +4. Install the database and activate it: Depending on your actual deployment architecture, you can choose from the following tutorials for installation and deployment: + + - Stand-Alone Deployment: [Stand-Alone Deployment ](../Deployment-and-Maintenance/Stand-Alone-Deployment_timecho.md) + + - Cluster Deployment:[Cluster Deployment](../Deployment-and-Maintenance/Cluster-Deployment_timecho.md) + + - Dual-Active Deployment:[Dual-Active Deployment](../Deployment-and-Maintenance/Dual-Active-Deployment_timecho.md) + +> ❗️Note: We currently still recommend direct installation and deployment on physical/virtual machines. For Docker deployment, please refer to [Docker Deployment](../Deployment-and-Maintenance/Docker-Deployment_timecho.md) + +5. Install supporting tools: The TimechoDB provides supporting tools such as a monitoring panel, which is recommended to be installed when deploying the TimechoDB to facilitate a more convenient usage: + + - Monitoring Panel: Provides hundreds of database monitoring metrics for fine-grained monitoring of IoTDB and its operating system, assisting in system optimization, performance tuning, and bottleneck detection. For installation steps, please see [Monitoring-panel Deployment ](../Deployment-and-Maintenance/Monitoring-panel-deployment.md) + + +## How to Use IoTDB? + +1. Database Modeling Design: Database modeling is a crucial step in creating a database system, involving the design of data structures and relationships to ensure that the organization of data meets the needs of specific applications. The following documents will help you quickly understand IoTDB's modeling design: + + - Introduction to Time Series Concepts: [Navigating Time Series Data](../Background-knowledge/Navigating_Time_Series_Data.md) + + - Introduction to Modeling Design:[Data Model and Terminology](../Background-knowledge/Data-Model-and-Terminology_timecho.md) + + - Introduction to Database:[Database Management](../Basic-Concept/Database-Management.md) + + - Introduction to Tables: [Table Management](../Basic-Concept/Table-Management.md) + +2. Data Insertion & Updates: IoTDB provides multiple methods for inserting real-time data. For basic data insertion and updating operations, please see [Write&Updata Data](../Basic-Concept/Write-Updata-Data.md) + +3. Data Querying: IoTDB offers a rich set of data querying capabilities. For a basic introduction to data querying, please see [Query Data](../Basic-Concept/Query-Data.md) + +4. Data Deletion: IoTDB supports two deletion methods: SQL-based deletion and automatic expiration deletion (TTL). + + - SQL-Based Deletion: For a basic introduction, please refer to [Delete Data](../Basic-Concept/Delete-Data.md) + + - Automatic Expiration Deletion (TTL): For a basic introduction, please see [TTL Delete Data](../Basic-Concept/TTL-Delete-Data.md) + +5. Advanced Features: In addition to common database functions such as insertion and querying, IoTDB also supports features like "data synchronization." For specific usage methods, please refer to the respective documents: + + - Data Synchronization: [Data Sync](../User-Manual/Data-Sync_timecho.md) + +6. Application Programming Interfaces (APIs): IoTDB provides various application programming interfaces (APIs) to facilitate developers' interaction with IoTDB in applications. Currently supported interfaces include [Java Native API](../API/Programming-Java-Native-API.md)、[Python Native API](../API/Programming-Python-Native-API.md)、[JDBC](../API/Programming-JDBC.md), and more. For more programming interfaces, please refer to the [Application Programming Interfaces] section on the official website. + +## What other convenient peripheral tools are available? + +In addition to its own rich features, IoTDB is complemented by a comprehensive suite of peripheral tools. This guide will help you quickly understand and use these tools: + + - Monitoring Panel: A tool for detailed monitoring of IoTDB and its operating system, covering hundreds of database monitoring indicators such as database performance and system resources, helping system optimization and bottleneck identification. For a detailed usage introduction, please refer to [Monitor Tool](../Tools-System/Monitor-Tool_timecho.md) + + +## Want to learn more technical details? + +If you want to explore IoTDB’s internal mechanisms further, refer to the following documents: + + - Data Partitioning and Load Balancing: IoTDB is designed with a partitioning strategy and load balancing algorithm to enhance cluster availability and performance. For more details, please see [Cluster data partitioning](../Technical-Insider/Cluster-data-partitioning.md) + + - Compression & Encoding: IoTDB employs various encoding and compression techniques optimized for different data types to improve storage efficiency. For more details, please see [Encoding and Compression](../Technical-Insider/Encoding-and-Compression.md) + + diff --git a/src/zh/UserGuide/Master/Table/QuickStart/QuickStart_timecho.md b/src/zh/UserGuide/Master/Table/QuickStart/QuickStart_timecho.md index fb883bfd6..83b6eab7c 100644 --- a/src/zh/UserGuide/Master/Table/QuickStart/QuickStart_timecho.md +++ b/src/zh/UserGuide/Master/Table/QuickStart/QuickStart_timecho.md @@ -79,7 +79,7 @@ IoTDB 除了自身拥有丰富的功能外，其周边的工具体系包含的种类十分齐全。本篇文档将帮助您快速使用周边工具体系： - - 监控面板：是一个对 IoTDB 及其所在操作系统进行细致监控的工具，涵盖数据库性能、系统资源等上百个数据库监控指标，助力系统优化与瓶颈识别等，具体使用介绍请查看 [监控面板部署](../Deployment-and-Maintenance/Monitoring-panel-deployment.md) + - 监控面板：是一个对 IoTDB 及其所在操作系统进行细致监控的工具，涵盖数据库性能、系统资源等上百个数据库监控指标，助力系统优化与瓶颈识别等，具体使用介绍请查看 [监控面板部署](../Tools-System/Monitor-Tool_timecho.md) ## 想了解更多技术细节？ diff --git a/src/zh/UserGuide/latest-Table/QuickStart/QuickStart_timecho.md b/src/zh/UserGuide/latest-Table/QuickStart/QuickStart_timecho.md index fb883bfd6..83b6eab7c 100644 --- a/src/zh/UserGuide/latest-Table/QuickStart/QuickStart_timecho.md +++ b/src/zh/UserGuide/latest-Table/QuickStart/QuickStart_timecho.md @@ -79,7 +79,7 @@ IoTDB 除了自身拥有丰富的功能外，其周边的工具体系包含的种类十分齐全。本篇文档将帮助您快速使用周边工具体系： - - 监控面板：是一个对 IoTDB 及其所在操作系统进行细致监控的工具，涵盖数据库性能、系统资源等上百个数据库监控指标，助力系统优化与瓶颈识别等，具体使用介绍请查看 [监控面板部署](../Deployment-and-Maintenance/Monitoring-panel-deployment.md) + - 监控面板：是一个对 IoTDB 及其所在操作系统进行细致监控的工具，涵盖数据库性能、系统资源等上百个数据库监控指标，助力系统优化与瓶颈识别等，具体使用介绍请查看 [监控面板部署](../Tools-System/Monitor-Tool_timecho.md) ## 想了解更多技术细节？ From 9779c4a024b380e48633adde53120000c0658f69 Mon Sep 17 00:00:00 2001 From: W1y1r <2730956796@qq.com> Date: Thu, 20 Feb 2025 19:06:50 +0800 Subject: [PATCH 2/4] Data synchronization error content --- .../Table/User-Manual/Data-Sync_apache.md | 521 ++++++++++++++++++ .../Table/User-Manual/Data-Sync_timecho.md | 113 ++-- .../User-Manual/Data-Sync_apache.md | 521 ++++++++++++++++++ .../User-Manual/Data-Sync_timecho.md | 113 ++-- .../Table/User-Manual/Data-Sync_apache.md | 4 +- .../User-Manual/Data-Sync_apache.md | 4 +- 6 files changed, 1194 insertions(+), 82 deletions(-) create mode 100644 src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md create mode 100644 src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md new file mode 100644 index 000000000..d6d84b652 --- /dev/null +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -0,0 +1,521 @@ + + +# Data Synchronization + +Data synchronization is a typical requirement in the Industrial Internet of Things (IIoT). Through data synchronization mechanisms, data sharing between IoTDB instances can be achieved, enabling the establishment of a complete data pipeline to meet needs such as internal and external network data exchange, edge-to-cloud synchronization, data migration, and data backup. + +## 1 Functional Overview + +### 1.1 Data Synchronization + +A data synchronization task consists of three stages: + +![](/img/en_dataSync01.png) + +- Source Stage: This stage is used to extract data from the source IoTDB, defined in the `source` section of the SQL statement. +- Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement. +- Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement. + +By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. + +### 1.2 Functional Limitations and Notes + +- Supports data synchronization from IoTDB version 1.x series to version 2.x and later. +- Does not support data synchronization from IoTDB version 2.x series to version 1.x series. +- When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. + +## 2 Usage Instructions + +A data synchronization task can be in one of three states: RUNNING, STOPPED, and DROPPED. The state transitions of the task are illustrated in the diagram below: + +![](/img/Data-Sync02.png) + +After creation, the task will start directly. Additionally, if the task stops due to an exception, the system will automatically attempt to restart it. + +We provide the following SQL statements for managing the state of synchronization tasks. + +### 2.1 Create a Task + +Use the `CREATE PIPE` statement to create a data synchronization task. Among the following attributes, `PipeId` and `sink` are required, while `source` and `processor` are optional. Note that the order of the `SOURCE` and `SINK` plugins cannot be swapped when writing the SQL. + +SQL Example: + +```SQL +CREATE PIPE [IF NOT EXISTS] -- PipeId is a unique name identifying the task +-- Data extraction plugin (optional) +WITH SOURCE ( + [ = ,], +) +-- Data processing plugin (optional) +WITH PROCESSOR ( + [ = ,], +) +-- Data transmission plugin (required) +WITH SINK ( + [ = ,], +) +``` + +**IF NOT EXISTS Semantics**: Ensures that the creation command is executed only if the specified Pipe does not exist, preventing errors caused by attempting to create an already existing Pipe. + + +### 2.2 Start a Task + +After creation, the task directly enters the RUNNING state and does not require manual startup. However, if the task is stopped using the `STOP PIPE` statement, you need to manually start it using the `START PIPE` statement. If the task stops due to an exception, it will automatically restart to resume data processing: + +```SQL +START PIPE +``` + +### 2.3 Stop a Task + +To stop data processing: + +```SQL +STOP PIPE +``` + +### 2.4 Delete a Task + +To delete a specified task: + +```SQL +DROP PIPE [IF EXISTS] +``` + +**IF EXISTS Semantics**: Ensures that the deletion command is executed only if the specified Pipe exists, preventing errors caused by attempting to delete a non-existent Pipe. + +**Note**: Deleting a task does not require stopping the synchronization task first. + +### 2.5 View Tasks + +To view all tasks: + +```SQL +SHOW PIPES +``` + +To view a specific task: + +```SQL +SHOW PIPE +``` + +Example Output of `SHOW PIPES`: + +```SQL ++--------------------------------+-----------------------+-------+----------+-------------+-----------------------------------------------------------+----------------+-------------------+-------------------------+ +| ID| CreationTime| State|PipeSource|PipeProcessor| PipeSink|ExceptionMessage|RemainingEventCount|EstimatedRemainingSeconds| ++--------------------------------+-----------------------+-------+----------+-------------+-----------------------------------------------------------+----------------+-------------------+-------------------------+ +|59abf95db892428b9d01c5fa318014ea|2024-06-17T14:03:44.189|RUNNING| {}| {}|{sink=iotdb-thrift-sink, sink.ip=127.0.0.1, sink.port=6668}| | 128| 1.03| ++--------------------------------+-----------------------+-------+----------+-------------+-----------------------------------------------------------+----------------+-------------------+-------------------------+ +``` + +**Column Descriptions**: + +- **ID**: Unique identifier of the synchronization task. +- **CreationTime**: Time when the task was created. +- **State**: Current state of the task. +- **PipeSource**: Source of the data stream. +- **PipeProcessor**: Processing logic applied during data transmission. +- **PipeSink**: Destination of the data stream. +- **ExceptionMessage**: Displays exception information for the task. +- **RemainingEventCount** (statistics may have delays): Number of remaining events, including data and metadata synchronization events, as well as system and user-defined events. +- **EstimatedRemainingSeconds** (statistics may have delays): Estimated remaining time to complete the transmission based on the current event count and pipe processing rate. + +### 2.6 Synchronization Plugins + +To make the architecture more flexible and adaptable to different synchronization scenarios, IoTDB supports plugin assembly in the synchronization task framework. The system provides some common pre-installed plugins, and you can also customize `processor` and `sink` plugins and load them into the IoTDB system. + +To view the plugins available in the system (including custom and built-in plugins), use the following statement: + +```SQL +SHOW PIPEPLUGINS +``` + +Example Output: + +```SQL +IoTDB> SHOW PIPEPLUGINS ++------------------------------+----------+--------------------------------------------------------------------------------------------------+----------------------------------------------------+ +| PluginName|PluginType| ClassName| PluginJar| ++------------------------------+----------+--------------------------------------------------------------------------------------------------+----------------------------------------------------+ +| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.processor.donothing.DoNothingProcessor| | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.connector.donothing.DoNothingConnector| | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.extractor.iotdb.IoTDBExtractor| | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | +| IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | ++------------------------------+----------+--------------------------------------------------------------------------------------------------+----------------------------------------------------+ + +``` + +Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](#reference-parameter-description): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeCustom PluginPlugin NameDescription
Source PluginNot Supportediotdb-sourceDefault extractor plugin for extracting historical or real-time data from IoTDB.
processor PluginSupporteddo-nothing-processorDefault processor plugin that does not process incoming data.
sink PluginSupporteddo-nothing-sinkDoes not process outgoing data.
iotdb-thrift-sinkDefault sink plugin for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded async non-blocking IO model, ideal for distributed target scenarios.
iotdb-thrift-ssl-sinkUsed for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded sync blocking IO model, suitable for high-security scenarios.
+ + +## 3 Usage Examples + +### 3.1 Full Data Synchronization + +This example demonstrates synchronizing all data from one IoTDB to another. The data pipeline is shown below: + +![](/img/e1.png) + +In this example, we create a synchronization task named `A2B` to synchronize all data from IoTDB A to IoTDB B. The `iotdb-thrift-sink` plugin (built-in) is used, and the `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB. + +SQL Example: + +```SQL +CREATE PIPE A2B +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB +) +``` + +### 3.2 Partial Data Synchronization + +This example demonstrates synchronizing data within a specific historical time range (from August 23, 2023, 8:00 to October 23, 2023, 8:00) to another IoTDB. The data pipeline is shown below: + +![](/img/e2.png) + +In this example, we create a synchronization task named `A2B`. First, we define the data range in the `source` configuration. Since we are synchronizing historical data (data that existed before the task was created), we need to configure the start time (`start-time`), end time (`end-time`), and the streaming mode (`mode.streaming`). The `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB. + +SQL Example: + +```SQL +CREATE PIPE A2B +WITH SOURCE ( + 'source' = 'iotdb-source', + 'mode.streaming' = 'true' -- Extraction mode for newly inserted data (after the pipe is created): + -- Whether to extract data in streaming mode (if set to false, batch mode is used). + 'start-time' = '2023.08.23T08:00:00+00:00', -- The event time at which data synchronization starts (inclusive). + 'end-time' = '2023.10.23T08:00:00+00:00' -- The event time at which data synchronization ends (inclusive). +) +WITH SINK ( + 'sink' = 'iotdb-thrift-async-sink', + 'node-urls' = '127.0.0.1:6668' -- The URL of the DataNode's data service port in the target IoTDB instance. +) +``` +### 3.3 Bidirectional Data Transmission + +This example demonstrates a scenario where two IoTDB instances act as dual-active systems. The data pipeline is shown below: + +![](/img/e3.png) + +To avoid infinite data loops, the `source.mode.double-living` parameter must be set to `true` on both IoTDB A and B, indicating that data forwarded from another pipe will not be retransmitted. + +SQL Example: On IoTDB A: + +```SQL +CREATE PIPE AB +WITH SOURCE ( + 'source.mode.double-living' = 'true' -- Do not forward data from other pipes +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB +) +``` + +On IoTDB B: + +```SQL +CREATE PIPE BA +WITH SOURCE ( + 'source.mode.double-living' = 'true' -- Do not forward data from other pipes +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6667' -- URL of the DataNode service port on the target IoTDB +) +``` + +### 3.4 Edge-to-Cloud Data Transmission + +This example demonstrates synchronizing data from multiple IoTDB clusters (B, C, D) to a central IoTDB cluster (A). The data pipeline is shown below: + +![](/img/sync_en_03.png) + +To synchronize data from clusters B, C, and D to cluster A, the `database-name` and `table-name` parameters are used to restrict the data range. + +SQL Example: On IoTDB B: + +```SQL +CREATE PIPE BA +WITH SOURCE ( + 'database-name' = 'db_b.*', -- Restrict the database scope + 'table-name' = '.*' -- Match all tables +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6667' -- URL of the DataNode service port on the target IoTDB +) +``` + +On IoTDB C ： + +```SQL +CREATE PIPE CA +WITH SOURCE ( + 'database-name' = 'db_c.*', -- Restrict the database scope + 'table-name' = '.*' -- Match all tables +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB +) +``` + +On IoTDB D： + +```SQL +CREATE PIPE DA +WITH SOURCE ( + 'database-name' = 'db_d.*', -- Restrict the database scope + 'table-name' = '.*' -- Match all tables +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6669' -- URL of the DataNode service port on the target IoTDB +) +``` + +### 3.5 Cascaded Data Transmission + +This example demonstrates cascading data transmission from IoTDB A to IoTDB B and then to IoTDB C. The data pipeline is shown below: + +![](/img/sync_en_04.png) + +To synchronize data from cluster A to cluster C, the `source.mode.double-living` parameter is set to `true` in the pipe between B and C. + +SQL Example: On IoTDB A: + +```SQL +CREATE PIPE AB +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB +) +``` + +On IoTDB B: + +```SQL +CREATE PIPE BC +WITH SOURCE ( + 'source.mode.double-living' = 'true' -- Do not forward data from other pipes +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6669' -- URL of the DataNode service port on the target IoTDB +) +``` + + +### 3.6 Compressed Synchronization + +IoTDB supports specifying data compression methods during synchronization. The `compressor` parameter can be configured to enable real-time data compression and transmission. Supported algorithms include `snappy`, `gzip`, `lz4`, `zstd`, and `lzma2`. Multiple algorithms can be combined and applied in the configured order. The `rate-limit-bytes-per-second` parameter (supported in V1.3.3 and later) limits the maximum number of bytes transmitted per second (calculated after compression). If set to a value less than 0, there is no limit. + +**SQL Example**: + +```SQL +CREATE PIPE A2B +WITH SINK ( + 'node-urls' = '127.0.0.1:6668', -- URL of the DataNode service port on the target IoTDB + 'compressor' = 'snappy,lz4', -- Compression algorithms + 'rate-limit-bytes-per-second' = '1048576' -- Maximum bytes allowed per second +) +``` + + +### 3.7 Encrypted Synchronization + +IoTDB supports SSL encryption during synchronization to securely transmit data between IoTDB instances. By configuring SSL-related parameters such as the certificate path (`ssl.trust-store-path`) and password (`ssl.trust-store-pwd`), data can be protected by SSL encryption during synchronization. + +**SQL Example**: + +```SQL +CREATE PIPE A2B +WITH SINK ( + 'sink' = 'iotdb-thrift-ssl-sink', + 'node-urls' = '127.0.0.1:6667', -- URL of the DataNode service port on the target IoTDB + 'ssl.trust-store-path' = 'pki/trusted', -- Path to the trust store certificate + 'ssl.trust-store-pwd' = 'root' -- Password for the trust store certificate +) +``` + + +## 4 Reference: Notes + +You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows: + + +```Properties +# pipe_receiver_file_dir +# If this property is unset, system will save the data in the default relative path directory under the IoTDB folder(i.e., %IOTDB_HOME%/${cn_system_dir}/pipe/receiver). +# If it is absolute, system will save the data in the exact location it points to. +# If it is relative, system will save the data in the relative path directory it indicates under the IoTDB folder. +# Note: If pipe_receiver_file_dir is assigned an empty string(i.e.,zero-size), it will be handled as a relative path. +# effectiveMode: restart +# For windows platform +# If its prefix is a drive specifier followed by "\\", or if its prefix is "\\\\", then the path is absolute. Otherwise, it is relative. +# pipe_receiver_file_dir=data\\confignode\\system\\pipe\\receiver +# For Linux platform +# If its prefix is "/", then the path is absolute. Otherwise, it is relative. +pipe_receiver_file_dir=data/confignode/system/pipe/receiver + +#################### +### Pipe Configuration +#################### + +# Uncomment the following field to configure the pipe lib directory. +# effectiveMode: first_start +# For Windows platform +# If its prefix is a drive specifier followed by "\\", or if its prefix is "\\\\", then the path is +# absolute. Otherwise, it is relative. +# pipe_lib_dir=ext\\pipe +# For Linux platform +# If its prefix is "/", then the path is absolute. Otherwise, it is relative. +pipe_lib_dir=ext/pipe + +# The maximum number of threads that can be used to execute the pipe subtasks in PipeSubtaskExecutor. +# The actual value will be min(pipe_subtask_executor_max_thread_num, max(1, CPU core number / 2)). +# effectiveMode: restart +# Datatype: int +pipe_subtask_executor_max_thread_num=5 + +# The connection timeout (in milliseconds) for the thrift client. +# effectiveMode: restart +# Datatype: int +pipe_sink_timeout_ms=900000 + +# The maximum number of selectors that can be used in the sink. +# Recommend to set this value to less than or equal to pipe_sink_max_client_number. +# effectiveMode: restart +# Datatype: int +pipe_sink_selector_number=4 + +# The maximum number of clients that can be used in the sink. +# effectiveMode: restart +# Datatype: int +pipe_sink_max_client_number=16 + +# The total bytes that all pipe sinks can transfer per second. +# When given a value less than or equal to 0, it means no limit. +# default value is -1, which means no limit. +# effectiveMode: hot_reload +# Datatype: double +pipe_all_sinks_rate_limit_bytes_per_second=-1 +``` + +## 5 Reference: Parameter Description + +### 5.1 source parameter + +| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | +| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- | +| source | iotdb-source | String: iotdb-source | Yes | - | +| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | +| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | +| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | +| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | +| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| forwarding-pipe-requests | Specifies whether to forward data that was synchronized via the pipe to external clusters. Typically used for setting up **active-active clusters**. In active-active cluster mode, this parameter should be set to `false` to prevent **infinite circular synchronization**. | Boolean: true / false | No | true | + +> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` +> +> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput. +> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput. + +### 5.2 sink parameter + +#### 5.2.1 iotdb-thrift-sink + +| **Parameter** | **Description** | Value Range | Required | Default Value | +| :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | + +#### 5.2.2 iotdb-thrift-ssl-sink + + +| **Parameter** | **Description** | Value Range | Required | Default Value | +| :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | \ No newline at end of file diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index 90bc5bc4f..705f3ba4c 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -18,12 +18,13 @@ under the License. --> +# Data Synchronization Data synchronization is a typical requirement in the Industrial Internet of Things (IIoT). Through data synchronization mechanisms, data sharing between IoTDB instances can be achieved, enabling the establishment of a complete data pipeline to meet needs such as internal and external network data exchange, edge-to-cloud synchronization, data migration, and data backup. -# Functional Overview +## 1 Functional Overview -## Data Synchronization +### 1.1 Data Synchronization A data synchronization task consists of three stages: @@ -35,13 +36,13 @@ A data synchronization task consists of three stages: By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. -## Functional Limitations and Notes +### 1.2 Functional Limitations and Notes - Supports data synchronization from IoTDB version 1.x series to version 2.x and later. - Does not support data synchronization from IoTDB version 2.x series to version 1.x series. - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. -# Usage Instructions +## 2 Usage Instructions A data synchronization task can be in one of three states: RUNNING, STOPPED, and DROPPED. The state transitions of the task are illustrated in the diagram below: @@ -51,7 +52,7 @@ After creation, the task will start directly. Additionally, if the task stops du We provide the following SQL statements for managing the state of synchronization tasks. -## Create a Task +### 2.1 Create a Task Use the `CREATE PIPE` statement to create a data synchronization task. Among the following attributes, `PipeId` and `sink` are required, while `source` and `processor` are optional. Note that the order of the `SOURCE` and `SINK` plugins cannot be swapped when writing the SQL. @@ -75,7 +76,7 @@ WITH SINK ( **IF NOT EXISTS Semantics**: Ensures that the creation command is executed only if the specified Pipe does not exist, preventing errors caused by attempting to create an already existing Pipe. -## Start a Task +### 2.2 Start a Task After creation, the task directly enters the RUNNING state and does not require manual startup. However, if the task is stopped using the `STOP PIPE` statement, you need to manually start it using the `START PIPE` statement. If the task stops due to an exception, it will automatically restart to resume data processing: @@ -83,7 +84,7 @@ After creation, the task directly enters the RUNNING state and does not require START PIPE ``` -## Stop a Task +### 2.3 Stop a Task To stop data processing: @@ -91,7 +92,7 @@ To stop data processing: STOP PIPE ``` -## Delete a Task +### 2.4 Delete a Task To delete a specified task: @@ -99,9 +100,11 @@ To delete a specified task: DROP PIPE [IF EXISTS] ``` -**IF EXISTS Semantics**: Ensures that the deletion command is executed only if the specified Pipe exists, preventing errors caused by attempting to delete a non-existent Pipe. **Note**: Deleting a task does not require stopping the synchronization task first. +**IF EXISTS Semantics**: Ensures that the deletion command is executed only if the specified Pipe exists, preventing errors caused by attempting to delete a non-existent Pipe. -## View Tasks +**Note**: Deleting a task does not require stopping the synchronization task first. + +### 2.5 View Tasks To view all tasks: @@ -137,7 +140,7 @@ Example Output of `SHOW PIPES`: - **RemainingEventCount** (statistics may have delays): Number of remaining events, including data and metadata synchronization events, as well as system and user-defined events. - **EstimatedRemainingSeconds** (statistics may have delays): Estimated remaining time to complete the transmission based on the current event count and pipe processing rate. -## Synchronization Plugins +### 2.6 Synchronization Plugins To make the architecture more flexible and adaptable to different synchronization scenarios, IoTDB supports plugin assembly in the synchronization task framework. The system provides some common pre-installed plugins, and you can also customize `processor` and `sink` plugins and load them into the IoTDB system. @@ -163,20 +166,52 @@ IoTDB> SHOW PIPEPLUGINS +------------------------------+----------+--------------------------------------------------------------------------------------------------+----------------------------------------------------+ ``` -Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](../Reference/System-Config-Manual.md) section): - -| **Type** | **Custom Plugin** | **Plugin Name** | **Description** | -| :---------------------- | :----------------------------------------------------------- | :--------------------- | :----------------------------------------------------------- | -| Source Plugin | Not Supported | `iotdb-source` | Default extractor plugin for extracting historical or real-time data from IoTDB. | -| Processor Plugin | Supported | `do-nothing-processor` | Default processor plugin that does not process incoming data. | -| Sink Plugin | Supported | `do-nothing-sink` | Does not process outgoing data. | -| `iotdb-thrift-sink` | Default sink plugin for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded async non-blocking IO model, ideal for distributed target scenarios. | | | -| `iotdb-air-gap-sink` | Used for cross-unidirectional data gate synchronization between IoTDB instances (V2.0.0+). Supports gate models like NARI Syskeeper 2000. | | | -| `iotdb-thrift-ssl-sink` | Used for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded sync blocking IO model, suitable for high-security scenarios. | | | - -# Usage Examples - -## Full Data Synchronization +Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](#reference-parameter-description): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeCustom PluginPlugin NameDescription
Source PluginNot Supportediotdb-sourceDefault extractor plugin for extracting historical or real-time data from IoTDB.
processor PluginSupporteddo-nothing-processorDefault processor plugin that does not process incoming data.
sink PluginSupporteddo-nothing-sinkDoes not process outgoing data.
iotdb-thrift-sinkDefault sink plugin for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded async non-blocking IO model, ideal for distributed target scenarios.
iotdb-air-gap-sinkUsed for cross-unidirectional data gate synchronization between IoTDB instances (V2.0.0+). Supports gate models like NARI Syskeeper 2000.
iotdb-thrift-ssl-sinkUsed for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded sync blocking IO model, suitable for high-security scenarios.
+ +## 3 Usage Examples + +### 3.1 Full Data Synchronization This example demonstrates synchronizing all data from one IoTDB to another. The data pipeline is shown below: @@ -194,7 +229,7 @@ WITH SINK ( ) ``` -## Partial Data Synchronization +### 3.2 Partial Data Synchronization This example demonstrates synchronizing data within a specific historical time range (from August 23, 2023, 8:00 to October 23, 2023, 8:00) to another IoTDB. The data pipeline is shown below: @@ -219,7 +254,7 @@ WITH SINK ( ) ``` -## Bidirectional Data Transmission +### 3.3 Bidirectional Data Transmission This example demonstrates a scenario where two IoTDB instances act as dual-active systems. The data pipeline is shown below: @@ -253,7 +288,7 @@ WITH SINK ( ) ``` -## Edge-to-Cloud Data Transmission +### 3.4 Edge-to-Cloud Data Transmission This example demonstrates synchronizing data from multiple IoTDB clusters (B, C, D) to a central IoTDB cluster (A). The data pipeline is shown below: @@ -303,7 +338,7 @@ WITH SINK ( ) ``` -## Cascaded Data Transmission +### 3.5 Cascaded Data Transmission This example demonstrates cascading data transmission from IoTDB A to IoTDB B and then to IoTDB C. The data pipeline is shown below: @@ -334,7 +369,7 @@ WITH SINK ( ) ``` -## Air-Gapped Data Transmission +### 3.6 Air-Gapped Data Transmission This example demonstrates synchronizing data from one IoTDB to another through a unidirectional air gap. The data pipeline is shown below: @@ -352,7 +387,7 @@ WITH SINK ( ) ``` -## Compressed Synchronization +### 3.7 Compressed Synchronization IoTDB supports specifying data compression methods during synchronization. The `compressor` parameter can be configured to enable real-time data compression and transmission. Supported algorithms include `snappy`, `gzip`, `lz4`, `zstd`, and `lzma2`. Multiple algorithms can be combined and applied in the configured order. The `rate-limit-bytes-per-second` parameter (supported in V1.3.3 and later) limits the maximum number of bytes transmitted per second (calculated after compression). If set to a value less than 0, there is no limit. @@ -367,7 +402,7 @@ WITH SINK ( ) ``` -## Encrypted Synchronization +### 3.8 Encrypted Synchronization IoTDB supports SSL encryption during synchronization to securely transmit data between IoTDB instances. By configuring SSL-related parameters such as the certificate path (`ssl.trust-store-path`) and password (`ssl.trust-store-pwd`), data can be protected by SSL encryption during synchronization. @@ -383,7 +418,7 @@ WITH SINK ( ) ``` -# Reference: Notes +## 4 Reference: Notes You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows: @@ -456,9 +491,9 @@ pipe_air_gap_receiver_port=9780 pipe_all_sinks_rate_limit_bytes_per_second=-1 ``` -# Reference: Parameter Description +## 5 Reference: Parameter Description -## S**ource** **p****arameter****s** +### 5.1 source parameter | **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | | :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- | @@ -477,9 +512,9 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 > - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput. > - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput. -## Sink **p****arameter****s** +### 5.2 sink parameter -#### iotdb-thrift-sink +#### 5.2.1 iotdb-thrift-sink | **Parameter** | **Description** | Value Range | Required | Default Value | | :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | @@ -494,7 +529,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | | rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -#### iotdb-air-gap-sink +#### 5.2.2 iotdb-air-gap-sink | **Parameter** | **Description** | Value Range | Required | Default Value | | :--------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | @@ -507,7 +542,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | | air-gap.handshake-timeout-ms | The timeout duration for the handshake requests when the sender and receiver attempt to establish a connection for the first time, in milliseconds. | Integer | No | 5000 | -#### iotdb-thrift-ssl-sink +#### 5.2.3 iotdb-thrift-ssl-sink | **Parameter** | **Description** | Value Range | Required | Default Value | | :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md new file mode 100644 index 000000000..d6d84b652 --- /dev/null +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -0,0 +1,521 @@ + + +# Data Synchronization + +Data synchronization is a typical requirement in the Industrial Internet of Things (IIoT). Through data synchronization mechanisms, data sharing between IoTDB instances can be achieved, enabling the establishment of a complete data pipeline to meet needs such as internal and external network data exchange, edge-to-cloud synchronization, data migration, and data backup. + +## 1 Functional Overview + +### 1.1 Data Synchronization + +A data synchronization task consists of three stages: + +![](/img/en_dataSync01.png) + +- Source Stage: This stage is used to extract data from the source IoTDB, defined in the `source` section of the SQL statement. +- Process Stage: This stage is used to process the data extracted from the source IoTDB, defined in the `processor` section of the SQL statement. +- Sink Stage: This stage is used to send data to the target IoTDB, defined in the `sink` section of the SQL statement. + +By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. + +### 1.2 Functional Limitations and Notes + +- Supports data synchronization from IoTDB version 1.x series to version 2.x and later. +- Does not support data synchronization from IoTDB version 2.x series to version 1.x series. +- When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. + +## 2 Usage Instructions + +A data synchronization task can be in one of three states: RUNNING, STOPPED, and DROPPED. The state transitions of the task are illustrated in the diagram below: + +![](/img/Data-Sync02.png) + +After creation, the task will start directly. Additionally, if the task stops due to an exception, the system will automatically attempt to restart it. + +We provide the following SQL statements for managing the state of synchronization tasks. + +### 2.1 Create a Task + +Use the `CREATE PIPE` statement to create a data synchronization task. Among the following attributes, `PipeId` and `sink` are required, while `source` and `processor` are optional. Note that the order of the `SOURCE` and `SINK` plugins cannot be swapped when writing the SQL. + +SQL Example: + +```SQL +CREATE PIPE [IF NOT EXISTS] -- PipeId is a unique name identifying the task +-- Data extraction plugin (optional) +WITH SOURCE ( + [ = ,], +) +-- Data processing plugin (optional) +WITH PROCESSOR ( + [ = ,], +) +-- Data transmission plugin (required) +WITH SINK ( + [ = ,], +) +``` + +**IF NOT EXISTS Semantics**: Ensures that the creation command is executed only if the specified Pipe does not exist, preventing errors caused by attempting to create an already existing Pipe. + + +### 2.2 Start a Task + +After creation, the task directly enters the RUNNING state and does not require manual startup. However, if the task is stopped using the `STOP PIPE` statement, you need to manually start it using the `START PIPE` statement. If the task stops due to an exception, it will automatically restart to resume data processing: + +```SQL +START PIPE +``` + +### 2.3 Stop a Task + +To stop data processing: + +```SQL +STOP PIPE +``` + +### 2.4 Delete a Task + +To delete a specified task: + +```SQL +DROP PIPE [IF EXISTS] +``` + +**IF EXISTS Semantics**: Ensures that the deletion command is executed only if the specified Pipe exists, preventing errors caused by attempting to delete a non-existent Pipe. + +**Note**: Deleting a task does not require stopping the synchronization task first. + +### 2.5 View Tasks + +To view all tasks: + +```SQL +SHOW PIPES +``` + +To view a specific task: + +```SQL +SHOW PIPE +``` + +Example Output of `SHOW PIPES`: + +```SQL ++--------------------------------+-----------------------+-------+----------+-------------+-----------------------------------------------------------+----------------+-------------------+-------------------------+ +| ID| CreationTime| State|PipeSource|PipeProcessor| PipeSink|ExceptionMessage|RemainingEventCount|EstimatedRemainingSeconds| ++--------------------------------+-----------------------+-------+----------+-------------+-----------------------------------------------------------+----------------+-------------------+-------------------------+ +|59abf95db892428b9d01c5fa318014ea|2024-06-17T14:03:44.189|RUNNING| {}| {}|{sink=iotdb-thrift-sink, sink.ip=127.0.0.1, sink.port=6668}| | 128| 1.03| ++--------------------------------+-----------------------+-------+----------+-------------+-----------------------------------------------------------+----------------+-------------------+-------------------------+ +``` + +**Column Descriptions**: + +- **ID**: Unique identifier of the synchronization task. +- **CreationTime**: Time when the task was created. +- **State**: Current state of the task. +- **PipeSource**: Source of the data stream. +- **PipeProcessor**: Processing logic applied during data transmission. +- **PipeSink**: Destination of the data stream. +- **ExceptionMessage**: Displays exception information for the task. +- **RemainingEventCount** (statistics may have delays): Number of remaining events, including data and metadata synchronization events, as well as system and user-defined events. +- **EstimatedRemainingSeconds** (statistics may have delays): Estimated remaining time to complete the transmission based on the current event count and pipe processing rate. + +### 2.6 Synchronization Plugins + +To make the architecture more flexible and adaptable to different synchronization scenarios, IoTDB supports plugin assembly in the synchronization task framework. The system provides some common pre-installed plugins, and you can also customize `processor` and `sink` plugins and load them into the IoTDB system. + +To view the plugins available in the system (including custom and built-in plugins), use the following statement: + +```SQL +SHOW PIPEPLUGINS +``` + +Example Output: + +```SQL +IoTDB> SHOW PIPEPLUGINS ++------------------------------+----------+--------------------------------------------------------------------------------------------------+----------------------------------------------------+ +| PluginName|PluginType| ClassName| PluginJar| ++------------------------------+----------+--------------------------------------------------------------------------------------------------+----------------------------------------------------+ +| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.processor.donothing.DoNothingProcessor| | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.connector.donothing.DoNothingConnector| | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.extractor.iotdb.IoTDBExtractor| | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | +| IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | ++------------------------------+----------+--------------------------------------------------------------------------------------------------+----------------------------------------------------+ + +``` + +Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](#reference-parameter-description): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeCustom PluginPlugin NameDescription
Source PluginNot Supportediotdb-sourceDefault extractor plugin for extracting historical or real-time data from IoTDB.
processor PluginSupporteddo-nothing-processorDefault processor plugin that does not process incoming data.
sink PluginSupporteddo-nothing-sinkDoes not process outgoing data.
iotdb-thrift-sinkDefault sink plugin for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded async non-blocking IO model, ideal for distributed target scenarios.
iotdb-thrift-ssl-sinkUsed for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded sync blocking IO model, suitable for high-security scenarios.
+ + +## 3 Usage Examples + +### 3.1 Full Data Synchronization + +This example demonstrates synchronizing all data from one IoTDB to another. The data pipeline is shown below: + +![](/img/e1.png) + +In this example, we create a synchronization task named `A2B` to synchronize all data from IoTDB A to IoTDB B. The `iotdb-thrift-sink` plugin (built-in) is used, and the `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB. + +SQL Example: + +```SQL +CREATE PIPE A2B +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB +) +``` + +### 3.2 Partial Data Synchronization + +This example demonstrates synchronizing data within a specific historical time range (from August 23, 2023, 8:00 to October 23, 2023, 8:00) to another IoTDB. The data pipeline is shown below: + +![](/img/e2.png) + +In this example, we create a synchronization task named `A2B`. First, we define the data range in the `source` configuration. Since we are synchronizing historical data (data that existed before the task was created), we need to configure the start time (`start-time`), end time (`end-time`), and the streaming mode (`mode.streaming`). The `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB. + +SQL Example: + +```SQL +CREATE PIPE A2B +WITH SOURCE ( + 'source' = 'iotdb-source', + 'mode.streaming' = 'true' -- Extraction mode for newly inserted data (after the pipe is created): + -- Whether to extract data in streaming mode (if set to false, batch mode is used). + 'start-time' = '2023.08.23T08:00:00+00:00', -- The event time at which data synchronization starts (inclusive). + 'end-time' = '2023.10.23T08:00:00+00:00' -- The event time at which data synchronization ends (inclusive). +) +WITH SINK ( + 'sink' = 'iotdb-thrift-async-sink', + 'node-urls' = '127.0.0.1:6668' -- The URL of the DataNode's data service port in the target IoTDB instance. +) +``` +### 3.3 Bidirectional Data Transmission + +This example demonstrates a scenario where two IoTDB instances act as dual-active systems. The data pipeline is shown below: + +![](/img/e3.png) + +To avoid infinite data loops, the `source.mode.double-living` parameter must be set to `true` on both IoTDB A and B, indicating that data forwarded from another pipe will not be retransmitted. + +SQL Example: On IoTDB A: + +```SQL +CREATE PIPE AB +WITH SOURCE ( + 'source.mode.double-living' = 'true' -- Do not forward data from other pipes +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB +) +``` + +On IoTDB B: + +```SQL +CREATE PIPE BA +WITH SOURCE ( + 'source.mode.double-living' = 'true' -- Do not forward data from other pipes +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6667' -- URL of the DataNode service port on the target IoTDB +) +``` + +### 3.4 Edge-to-Cloud Data Transmission + +This example demonstrates synchronizing data from multiple IoTDB clusters (B, C, D) to a central IoTDB cluster (A). The data pipeline is shown below: + +![](/img/sync_en_03.png) + +To synchronize data from clusters B, C, and D to cluster A, the `database-name` and `table-name` parameters are used to restrict the data range. + +SQL Example: On IoTDB B: + +```SQL +CREATE PIPE BA +WITH SOURCE ( + 'database-name' = 'db_b.*', -- Restrict the database scope + 'table-name' = '.*' -- Match all tables +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6667' -- URL of the DataNode service port on the target IoTDB +) +``` + +On IoTDB C ： + +```SQL +CREATE PIPE CA +WITH SOURCE ( + 'database-name' = 'db_c.*', -- Restrict the database scope + 'table-name' = '.*' -- Match all tables +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB +) +``` + +On IoTDB D： + +```SQL +CREATE PIPE DA +WITH SOURCE ( + 'database-name' = 'db_d.*', -- Restrict the database scope + 'table-name' = '.*' -- Match all tables +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6669' -- URL of the DataNode service port on the target IoTDB +) +``` + +### 3.5 Cascaded Data Transmission + +This example demonstrates cascading data transmission from IoTDB A to IoTDB B and then to IoTDB C. The data pipeline is shown below: + +![](/img/sync_en_04.png) + +To synchronize data from cluster A to cluster C, the `source.mode.double-living` parameter is set to `true` in the pipe between B and C. + +SQL Example: On IoTDB A: + +```SQL +CREATE PIPE AB +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB +) +``` + +On IoTDB B: + +```SQL +CREATE PIPE BC +WITH SOURCE ( + 'source.mode.double-living' = 'true' -- Do not forward data from other pipes +) +WITH SINK ( + 'sink' = 'iotdb-thrift-sink', + 'node-urls' = '127.0.0.1:6669' -- URL of the DataNode service port on the target IoTDB +) +``` + + +### 3.6 Compressed Synchronization + +IoTDB supports specifying data compression methods during synchronization. The `compressor` parameter can be configured to enable real-time data compression and transmission. Supported algorithms include `snappy`, `gzip`, `lz4`, `zstd`, and `lzma2`. Multiple algorithms can be combined and applied in the configured order. The `rate-limit-bytes-per-second` parameter (supported in V1.3.3 and later) limits the maximum number of bytes transmitted per second (calculated after compression). If set to a value less than 0, there is no limit. + +**SQL Example**: + +```SQL +CREATE PIPE A2B +WITH SINK ( + 'node-urls' = '127.0.0.1:6668', -- URL of the DataNode service port on the target IoTDB + 'compressor' = 'snappy,lz4', -- Compression algorithms + 'rate-limit-bytes-per-second' = '1048576' -- Maximum bytes allowed per second +) +``` + + +### 3.7 Encrypted Synchronization + +IoTDB supports SSL encryption during synchronization to securely transmit data between IoTDB instances. By configuring SSL-related parameters such as the certificate path (`ssl.trust-store-path`) and password (`ssl.trust-store-pwd`), data can be protected by SSL encryption during synchronization. + +**SQL Example**: + +```SQL +CREATE PIPE A2B +WITH SINK ( + 'sink' = 'iotdb-thrift-ssl-sink', + 'node-urls' = '127.0.0.1:6667', -- URL of the DataNode service port on the target IoTDB + 'ssl.trust-store-path' = 'pki/trusted', -- Path to the trust store certificate + 'ssl.trust-store-pwd' = 'root' -- Password for the trust store certificate +) +``` + + +## 4 Reference: Notes + +You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows: + + +```Properties +# pipe_receiver_file_dir +# If this property is unset, system will save the data in the default relative path directory under the IoTDB folder(i.e., %IOTDB_HOME%/${cn_system_dir}/pipe/receiver). +# If it is absolute, system will save the data in the exact location it points to. +# If it is relative, system will save the data in the relative path directory it indicates under the IoTDB folder. +# Note: If pipe_receiver_file_dir is assigned an empty string(i.e.,zero-size), it will be handled as a relative path. +# effectiveMode: restart +# For windows platform +# If its prefix is a drive specifier followed by "\\", or if its prefix is "\\\\", then the path is absolute. Otherwise, it is relative. +# pipe_receiver_file_dir=data\\confignode\\system\\pipe\\receiver +# For Linux platform +# If its prefix is "/", then the path is absolute. Otherwise, it is relative. +pipe_receiver_file_dir=data/confignode/system/pipe/receiver + +#################### +### Pipe Configuration +#################### + +# Uncomment the following field to configure the pipe lib directory. +# effectiveMode: first_start +# For Windows platform +# If its prefix is a drive specifier followed by "\\", or if its prefix is "\\\\", then the path is +# absolute. Otherwise, it is relative. +# pipe_lib_dir=ext\\pipe +# For Linux platform +# If its prefix is "/", then the path is absolute. Otherwise, it is relative. +pipe_lib_dir=ext/pipe + +# The maximum number of threads that can be used to execute the pipe subtasks in PipeSubtaskExecutor. +# The actual value will be min(pipe_subtask_executor_max_thread_num, max(1, CPU core number / 2)). +# effectiveMode: restart +# Datatype: int +pipe_subtask_executor_max_thread_num=5 + +# The connection timeout (in milliseconds) for the thrift client. +# effectiveMode: restart +# Datatype: int +pipe_sink_timeout_ms=900000 + +# The maximum number of selectors that can be used in the sink. +# Recommend to set this value to less than or equal to pipe_sink_max_client_number. +# effectiveMode: restart +# Datatype: int +pipe_sink_selector_number=4 + +# The maximum number of clients that can be used in the sink. +# effectiveMode: restart +# Datatype: int +pipe_sink_max_client_number=16 + +# The total bytes that all pipe sinks can transfer per second. +# When given a value less than or equal to 0, it means no limit. +# default value is -1, which means no limit. +# effectiveMode: hot_reload +# Datatype: double +pipe_all_sinks_rate_limit_bytes_per_second=-1 +``` + +## 5 Reference: Parameter Description + +### 5.1 source parameter + +| **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | +| :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- | +| source | iotdb-source | String: iotdb-source | Yes | - | +| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | +| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | +| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | +| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | +| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| forwarding-pipe-requests | Specifies whether to forward data that was synchronized via the pipe to external clusters. Typically used for setting up **active-active clusters**. In active-active cluster mode, this parameter should be set to `false` to prevent **infinite circular synchronization**. | Boolean: true / false | No | true | + +> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` +> +> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput. +> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput. + +### 5.2 sink parameter + +#### 5.2.1 iotdb-thrift-sink + +| **Parameter** | **Description** | Value Range | Required | Default Value | +| :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | + +#### 5.2.2 iotdb-thrift-ssl-sink + + +| **Parameter** | **Description** | Value Range | Required | Default Value | +| :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | \ No newline at end of file diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index 90bc5bc4f..705f3ba4c 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -18,12 +18,13 @@ under the License. --> +# Data Synchronization Data synchronization is a typical requirement in the Industrial Internet of Things (IIoT). Through data synchronization mechanisms, data sharing between IoTDB instances can be achieved, enabling the establishment of a complete data pipeline to meet needs such as internal and external network data exchange, edge-to-cloud synchronization, data migration, and data backup. -# Functional Overview +## 1 Functional Overview -## Data Synchronization +### 1.1 Data Synchronization A data synchronization task consists of three stages: @@ -35,13 +36,13 @@ A data synchronization task consists of three stages: By declaratively configuring these three parts in an SQL statement, flexible data synchronization capabilities can be achieved. -## Functional Limitations and Notes +### 1.2 Functional Limitations and Notes - Supports data synchronization from IoTDB version 1.x series to version 2.x and later. - Does not support data synchronization from IoTDB version 2.x series to version 1.x series. - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. -# Usage Instructions +## 2 Usage Instructions A data synchronization task can be in one of three states: RUNNING, STOPPED, and DROPPED. The state transitions of the task are illustrated in the diagram below: @@ -51,7 +52,7 @@ After creation, the task will start directly. Additionally, if the task stops du We provide the following SQL statements for managing the state of synchronization tasks. -## Create a Task +### 2.1 Create a Task Use the `CREATE PIPE` statement to create a data synchronization task. Among the following attributes, `PipeId` and `sink` are required, while `source` and `processor` are optional. Note that the order of the `SOURCE` and `SINK` plugins cannot be swapped when writing the SQL. @@ -75,7 +76,7 @@ WITH SINK ( **IF NOT EXISTS Semantics**: Ensures that the creation command is executed only if the specified Pipe does not exist, preventing errors caused by attempting to create an already existing Pipe. -## Start a Task +### 2.2 Start a Task After creation, the task directly enters the RUNNING state and does not require manual startup. However, if the task is stopped using the `STOP PIPE` statement, you need to manually start it using the `START PIPE` statement. If the task stops due to an exception, it will automatically restart to resume data processing: @@ -83,7 +84,7 @@ After creation, the task directly enters the RUNNING state and does not require START PIPE ``` -## Stop a Task +### 2.3 Stop a Task To stop data processing: @@ -91,7 +92,7 @@ To stop data processing: STOP PIPE ``` -## Delete a Task +### 2.4 Delete a Task To delete a specified task: @@ -99,9 +100,11 @@ To delete a specified task: DROP PIPE [IF EXISTS] ``` -**IF EXISTS Semantics**: Ensures that the deletion command is executed only if the specified Pipe exists, preventing errors caused by attempting to delete a non-existent Pipe. **Note**: Deleting a task does not require stopping the synchronization task first. +**IF EXISTS Semantics**: Ensures that the deletion command is executed only if the specified Pipe exists, preventing errors caused by attempting to delete a non-existent Pipe. -## View Tasks +**Note**: Deleting a task does not require stopping the synchronization task first. + +### 2.5 View Tasks To view all tasks: @@ -137,7 +140,7 @@ Example Output of `SHOW PIPES`: - **RemainingEventCount** (statistics may have delays): Number of remaining events, including data and metadata synchronization events, as well as system and user-defined events. - **EstimatedRemainingSeconds** (statistics may have delays): Estimated remaining time to complete the transmission based on the current event count and pipe processing rate. -## Synchronization Plugins +### 2.6 Synchronization Plugins To make the architecture more flexible and adaptable to different synchronization scenarios, IoTDB supports plugin assembly in the synchronization task framework. The system provides some common pre-installed plugins, and you can also customize `processor` and `sink` plugins and load them into the IoTDB system. @@ -163,20 +166,52 @@ IoTDB> SHOW PIPEPLUGINS +------------------------------+----------+--------------------------------------------------------------------------------------------------+----------------------------------------------------+ ``` -Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](../Reference/System-Config-Manual.md) section): - -| **Type** | **Custom Plugin** | **Plugin Name** | **Description** | -| :---------------------- | :----------------------------------------------------------- | :--------------------- | :----------------------------------------------------------- | -| Source Plugin | Not Supported | `iotdb-source` | Default extractor plugin for extracting historical or real-time data from IoTDB. | -| Processor Plugin | Supported | `do-nothing-processor` | Default processor plugin that does not process incoming data. | -| Sink Plugin | Supported | `do-nothing-sink` | Does not process outgoing data. | -| `iotdb-thrift-sink` | Default sink plugin for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded async non-blocking IO model, ideal for distributed target scenarios. | | | -| `iotdb-air-gap-sink` | Used for cross-unidirectional data gate synchronization between IoTDB instances (V2.0.0+). Supports gate models like NARI Syskeeper 2000. | | | -| `iotdb-thrift-ssl-sink` | Used for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded sync blocking IO model, suitable for high-security scenarios. | | | - -# Usage Examples - -## Full Data Synchronization +Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](#reference-parameter-description): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeCustom PluginPlugin NameDescription
Source PluginNot Supportediotdb-sourceDefault extractor plugin for extracting historical or real-time data from IoTDB.
processor PluginSupporteddo-nothing-processorDefault processor plugin that does not process incoming data.
sink PluginSupporteddo-nothing-sinkDoes not process outgoing data.
iotdb-thrift-sinkDefault sink plugin for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded async non-blocking IO model, ideal for distributed target scenarios.
iotdb-air-gap-sinkUsed for cross-unidirectional data gate synchronization between IoTDB instances (V2.0.0+). Supports gate models like NARI Syskeeper 2000.
iotdb-thrift-ssl-sinkUsed for data transmission between IoTDB instances (V2.0.0+). Uses Thrift RPC framework with a multi-threaded sync blocking IO model, suitable for high-security scenarios.
+ +## 3 Usage Examples + +### 3.1 Full Data Synchronization This example demonstrates synchronizing all data from one IoTDB to another. The data pipeline is shown below: @@ -194,7 +229,7 @@ WITH SINK ( ) ``` -## Partial Data Synchronization +### 3.2 Partial Data Synchronization This example demonstrates synchronizing data within a specific historical time range (from August 23, 2023, 8:00 to October 23, 2023, 8:00) to another IoTDB. The data pipeline is shown below: @@ -219,7 +254,7 @@ WITH SINK ( ) ``` -## Bidirectional Data Transmission +### 3.3 Bidirectional Data Transmission This example demonstrates a scenario where two IoTDB instances act as dual-active systems. The data pipeline is shown below: @@ -253,7 +288,7 @@ WITH SINK ( ) ``` -## Edge-to-Cloud Data Transmission +### 3.4 Edge-to-Cloud Data Transmission This example demonstrates synchronizing data from multiple IoTDB clusters (B, C, D) to a central IoTDB cluster (A). The data pipeline is shown below: @@ -303,7 +338,7 @@ WITH SINK ( ) ``` -## Cascaded Data Transmission +### 3.5 Cascaded Data Transmission This example demonstrates cascading data transmission from IoTDB A to IoTDB B and then to IoTDB C. The data pipeline is shown below: @@ -334,7 +369,7 @@ WITH SINK ( ) ``` -## Air-Gapped Data Transmission +### 3.6 Air-Gapped Data Transmission This example demonstrates synchronizing data from one IoTDB to another through a unidirectional air gap. The data pipeline is shown below: @@ -352,7 +387,7 @@ WITH SINK ( ) ``` -## Compressed Synchronization +### 3.7 Compressed Synchronization IoTDB supports specifying data compression methods during synchronization. The `compressor` parameter can be configured to enable real-time data compression and transmission. Supported algorithms include `snappy`, `gzip`, `lz4`, `zstd`, and `lzma2`. Multiple algorithms can be combined and applied in the configured order. The `rate-limit-bytes-per-second` parameter (supported in V1.3.3 and later) limits the maximum number of bytes transmitted per second (calculated after compression). If set to a value less than 0, there is no limit. @@ -367,7 +402,7 @@ WITH SINK ( ) ``` -## Encrypted Synchronization +### 3.8 Encrypted Synchronization IoTDB supports SSL encryption during synchronization to securely transmit data between IoTDB instances. By configuring SSL-related parameters such as the certificate path (`ssl.trust-store-path`) and password (`ssl.trust-store-pwd`), data can be protected by SSL encryption during synchronization. @@ -383,7 +418,7 @@ WITH SINK ( ) ``` -# Reference: Notes +## 4 Reference: Notes You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows: @@ -456,9 +491,9 @@ pipe_air_gap_receiver_port=9780 pipe_all_sinks_rate_limit_bytes_per_second=-1 ``` -# Reference: Parameter Description +## 5 Reference: Parameter Description -## S**ource** **p****arameter****s** +### 5.1 source parameter | **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | | :----------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------- | :---------------------------------------------------------- | @@ -477,9 +512,9 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 > - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput. > - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput. -## Sink **p****arameter****s** +### 5.2 sink parameter -#### iotdb-thrift-sink +#### 5.2.1 iotdb-thrift-sink | **Parameter** | **Description** | Value Range | Required | Default Value | | :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | @@ -494,7 +529,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | | rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -#### iotdb-air-gap-sink +#### 5.2.2 iotdb-air-gap-sink | **Parameter** | **Description** | Value Range | Required | Default Value | | :--------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | @@ -507,7 +542,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | | air-gap.handshake-timeout-ms | The timeout duration for the handshake requests when the sender and receiver attempt to establish a connection for the first time, in milliseconds. | Integer | No | 5000 | -#### iotdb-thrift-ssl-sink +#### 5.2.3 iotdb-thrift-ssl-sink | **Parameter** | **Description** | Value Range | Required | Default Value | | :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- | :------------ | diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index bc4dbdf81..4843e88a1 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -92,7 +92,7 @@ START PIPE STOP PIPE ``` -### 2.4 删除任务 +### 2.4 删除任务 删除指定任务： @@ -140,7 +140,7 @@ SHOW PIPE - **RemainingEventCount（统计存在延迟）**：剩余 event 数，当前数据同步任务中的所有 event 总数，包括数据同步的 event，以及系统和用户自定义的 event。 - **EstimatedRemainingSeconds（统计存在延迟）**：剩余时间，基于当前 event 个数和 pipe 处速率，预估完成传输的剩余时间。 -### 同步插件 +### 2.6 同步插件 为了使得整体架构更加灵活以匹配不同的同步场景需求，我们支持在同步任务框架中进行插件组装。系统为您预置了一些常用插件可直接使用，同时您也可以自定义 processor 插件 和 Sink 插件，并加载至 IoTDB 系统进行使用。查看系统中的插件（含自定义与内置插件）可以用以下语句： diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index bc4dbdf81..4843e88a1 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -92,7 +92,7 @@ START PIPE STOP PIPE ``` -### 2.4 删除任务 +### 2.4 删除任务 删除指定任务： @@ -140,7 +140,7 @@ SHOW PIPE - **RemainingEventCount（统计存在延迟）**：剩余 event 数，当前数据同步任务中的所有 event 总数，包括数据同步的 event，以及系统和用户自定义的 event。 - **EstimatedRemainingSeconds（统计存在延迟）**：剩余时间，基于当前 event 个数和 pipe 处速率，预估完成传输的剩余时间。 -### 同步插件 +### 2.6 同步插件 为了使得整体架构更加灵活以匹配不同的同步场景需求，我们支持在同步任务框架中进行插件组装。系统为您预置了一些常用插件可直接使用，同时您也可以自定义 processor 插件 和 Sink 插件，并加载至 IoTDB 系统进行使用。查看系统中的插件（含自定义与内置插件）可以用以下语句： From ddc6e638cecb3ec45de7501e291d525d674e5937 Mon Sep 17 00:00:00 2001 From: W1y1r <2730956796@qq.com> Date: Thu, 20 Feb 2025 19:15:47 +0800 Subject: [PATCH 3/4] Enterprise Edition Sidebar --- .../sidebar_timecho/V2.0.x/en-Table.ts | 74 +++++++++++++++++-- 1 file changed, 69 insertions(+), 5 deletions(-) diff --git a/src/.vuepress/sidebar_timecho/V2.0.x/en-Table.ts b/src/.vuepress/sidebar_timecho/V2.0.x/en-Table.ts index c0caec281..81f57e5c8 100644 --- a/src/.vuepress/sidebar_timecho/V2.0.x/en-Table.ts +++ b/src/.vuepress/sidebar_timecho/V2.0.x/en-Table.ts @@ -22,6 +22,28 @@ export const enSidebar = { text: 'IoTDB User Guide', children: [], }, + { + text: 'About IoTDB', + collapsible: true, + prefix: 'IoTDB-Introduction/', + // children: 'structure', 使用该方式自动获取文件夹下的文件 + children: [ + { text: 'IoTDB Introduction', link: 'IoTDB-Introduction_timecho' }, + { text: 'Scenario', link: 'Scenario' }, + { text: 'Release history', link: 'Release-history_timecho' }, + ], + }, + { + text: 'Background knowledge', + collapsible: true, + prefix: 'Background-knowledge/', + children: [ + { text: 'Common Concepts', link: 'Cluster-Concept_timecho' }, + { text: 'Timeseries Data Model', link: 'Navigating_Time_Series_Data' }, + { text: 'Modeling Scheme Design', link: 'Data-Model-and-Terminology_timecho' }, + { text: 'Data Type', link: 'Data-Type' }, + ], + }, { text: 'Quick Start', link: 'QuickStart/QuickStart_timecho', @@ -74,16 +96,24 @@ export const enSidebar = { { text: 'TTL Delete Data', link: 'TTL-Delete-Data' }, ], }, + ], + }, ], }, - + { + text: 'Advanced Features', + collapsible: true, + prefix: 'User-Manual/', + // children: 'structure', + children: [ + { text: 'Data Sync', link: 'Data-Sync_timecho' }, + { text: 'Tiered Storage', link: 'Tiered-Storage_timecho' }, { - text: 'Maintenance SQL', + text: 'System Maintenance', collapsible: true, children: [ { text: 'Query Performance Analysis', link: 'Query-Performance-Analysis' }, - { text: 'Load Balance', link: 'Load-Balance' }, - { text: 'Data Recovery', link: 'Data-Recovery' }, + { text: 'Maintenance statement', link: 'Maintenance-statement' }, ], }, ], @@ -95,6 +125,21 @@ export const enSidebar = { // children: 'structure', children: [ { text: 'CLI', link: 'CLI' }, + { text: 'Monitor Tool', link: 'Monitor-Tool_timecho' }, + { text: 'Benchmark Tool', link: 'Benchmark' }, + { text: 'Cluster Management Tool', link: 'Maintenance-Tool_timecho' }, + ], + }, + { + text: 'API', + collapsible: true, + prefix: 'API/', + // children: 'structure', + children: [ + { text: 'Java Native API', link: 'Programming-Java-Native-API' }, + { text: 'Python Native API', link: 'Programming-Python-Native-API' }, + { text: 'JDBC', link: 'Programming-JDBC' }, + { text: 'RESTAPI V1 ', link: 'RestServiceV1' }, ], }, { @@ -120,6 +165,24 @@ export const enSidebar = { { text: 'LIMIT&OFFSET Clause', link: 'Limit-Offset-Clause' }, ], }, + { + text: 'Functions and Operators', + collapsible: true, + children: [ + { text: 'Basis Functions', link: 'Basis-Function' }, + { text: 'Featured Functions', link: 'Featured-Functions' }, + ], + }, + ], + }, + { + text: 'Technical Insider', + collapsible: true, + prefix: 'Technical-Insider/', + // children: 'structure', + children: [ + { text: 'Compression & Encoding', link: 'Encoding-and-Compression' }, + { text: 'Data Partitioning & Load Balancing', link: 'Cluster-data-partitioning' }, ], }, { @@ -128,8 +191,9 @@ export const enSidebar = { prefix: 'Reference/', // children: 'structure', children: [ - { text: 'Config Manual', link: 'System-Config-Manual' }, { text: 'Sample Data', link: 'Sample-Data' }, + { text: 'Config Manual', link: 'System-Config-Manual' }, + { text: 'Status Codes', link: 'Status-Codes' }, ], }, ], From 372546f21b194c855694ca6328a9853090d3d79e Mon Sep 17 00:00:00 2001 From: W1y1r <2730956796@qq.com> Date: Thu, 20 Feb 2025 19:20:09 +0800 Subject: [PATCH 4/4] Open source version sidebar (incomplete) --- src/.vuepress/sidebar/V2.0.x/en-Table.ts | 47 +++++++++++++++++++++--- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/src/.vuepress/sidebar/V2.0.x/en-Table.ts b/src/.vuepress/sidebar/V2.0.x/en-Table.ts index 45410f14b..4f27437c0 100644 --- a/src/.vuepress/sidebar/V2.0.x/en-Table.ts +++ b/src/.vuepress/sidebar/V2.0.x/en-Table.ts @@ -22,6 +22,17 @@ export const enSidebar = { text: 'IoTDB User Guide (V1.3.3)', children: [], }, + { + text: 'Background knowledge', + collapsible: true, + prefix: 'Background-knowledge/', + children: [ + { text: 'Common Concepts', link: 'Cluster-Concept_apache' }, + { text: 'Timeseries Data Model', link: 'Navigating_Time_Series_Data' }, + { text: 'Modeling Scheme Design', link: 'Data-Model-and-Terminology_apache' }, + { text: 'Data Type', link: 'Data-Type' }, + ], + }, { text: 'Quick Start', link: 'QuickStart/QuickStart_apache', @@ -74,16 +85,23 @@ export const enSidebar = { { text: 'TTL Delete Data', link: 'TTL-Delete-Data' }, ], }, + ], + }, ], }, - + { + text: 'Advanced Features', + collapsible: true, + prefix: 'User-Manual/', + // children: 'structure', + children: [ + { text: 'Data Sync', link: 'Data-Sync_apache' }, { - text: 'Maintenance SQL', + text: 'System Maintenance', collapsible: true, children: [ { text: 'Query Performance Analysis', link: 'Query-Performance-Analysis' }, - { text: 'Load Balance', link: 'Load-Balance' }, - { text: 'Data Recovery', link: 'Data-Recovery' }, + { text: 'Maintenance statement', link: 'Maintenance-statement' }, ], }, ], @@ -120,6 +138,24 @@ export const enSidebar = { { text: 'LIMIT&OFFSET Clause', link: 'Limit-Offset-Clause' }, ], }, + { + text: 'Functions and Operators', + collapsible: true, + children: [ + { text: 'Basis Functions', link: 'Basis-Function' }, + { text: 'Featured Functions', link: 'Featured-Functions' }, + ], + }, + ], + }, + { + text: 'Technical Insider', + collapsible: true, + prefix: 'Technical-Insider/', + // children: 'structure', + children: [ + { text: 'Compression & Encoding', link: 'Encoding-and-Compression' }, + { text: 'Data Partitioning & Load Balancing', link: 'Cluster-data-partitioning' }, ], }, { @@ -128,8 +164,9 @@ export const enSidebar = { prefix: 'Reference/', // children: 'structure', children: [ - { text: 'Config Manual', link: 'System-Config-Manual' }, { text: 'Sample Data', link: 'Sample-Data' }, + { text: 'Config Manual', link: 'System-Config-Manual' }, + { text: 'Status Codes', link: 'Status-Codes' }, ], }, ],