diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt index 0a32715f..a3a95307 100644 --- a/.github/actions/spelling/allow.txt +++ b/.github/actions/spelling/allow.txt @@ -751,4 +751,23 @@ paimon upserts resync resynced - +changeit +dedup +dedupe +deststoretype +DLT +DLTUSRPRF +ICERT +keystore +keytool +OWNOBJOPT +ssltls +storepass +storetype +SYSCOLUMNS +SYSINDEXES +SYSKEYCST +SYSTABLES +tls +tlsssl +truststore diff --git a/docs/connectors/supported-data-sources.md b/docs/connectors/supported-data-sources.md index 8774e05c..3627cee0 100644 --- a/docs/connectors/supported-data-sources.md +++ b/docs/connectors/supported-data-sources.md @@ -83,7 +83,7 @@ import TabItem from '@theme/TabItem'; Doris ✅ - ✅ + ➖ ➖ ✅ ✅ diff --git a/docs/data-replication/create-task.md b/docs/data-replication/create-task.md index 203048ae..f3ca1337 100644 --- a/docs/data-replication/create-task.md +++ b/docs/data-replication/create-task.md @@ -99,6 +99,8 @@ As an example of creating a data replication task, the article demonstrates the * **Data Writing Mode**: Select according to business needs. * **Process by Event Type**: If you choose this, you also need to select data writing strategies for insert, update, and delete events. * **Statistical Append Write**: Only processes insert events, discarding update and delete events. + * **Increment Exactly Once Write**: Creates a cache table on the target to mark and check events, ensuring each change is written exactly once. Recommended when association fields (join keys) may change; enabling can impact sync performance. The target database must permit table creation and data writes. + * **Time Window**: Sets the retention period for the cache used by exactly-once dedup checks (the validity window). Expired cache entries are cleaned up daily. Longer windows consume more storage on the target database. * **Data Source Exclusive Configuration**: Choose whether to save deleted data. * **Synchronize Partition Properties**: When this feature is enabled, TapData will automatically create a sharded collection in the target database. This function is only effective when both the source and target databases are MongoDB clusters. * **Data Model** @@ -123,4 +125,3 @@ As an example of creating a data replication task, the article demonstrates the ## See also [Monitor or Manage Tasks](../data-transformation/manage-task.md) - diff --git a/docs/data-replication/incremental-check.md b/docs/data-replication/incremental-check.md index 77bc167d..bf0a1d21 100644 --- a/docs/data-replication/incremental-check.md +++ b/docs/data-replication/incremental-check.md @@ -38,6 +38,10 @@ To address these challenges, TapData introduces **Incremental Data Validation**, - **Incremental Validation**: When enabled, TapData will sample and validate newly inserted or updated records to identify data inconsistencies during sync. - **Validation Frequency**: Controls how many records are validated per second. TapData will compare records by primary key to check for differences between source and target. Default is 10 records/sec. Adjust based on workload. + - **Table Filter**: Configure which tables to validate. Defaults to all tables. You can include or exclude specific tables, or use regular expressions to select tables for customized needs. + - **Time Precision**: Supports strict match, rounding, and truncation. Default is strict match, requiring time-field values to be identical between source and target. Rounding or truncation allows small tolerances where exact precision is not required. + - **Queue Capacity**: Default is Auto. The system dynamically adjusts buffer size based on task concurrency and validation rate. Increasing it manually may raise resource consumption. + - **Check No Primary Key Tables**: Disabled by default—only tables with a primary key are validated. If the source or target has tables without a primary key, define a key first or enable this option to include them. - **Auto Repair**: When enabled, TapData will re-fetch and re-sync inconsistent records from the source, ensuring consistency without manual fixes. 5. Before starting the task, it is recommended to go to **Settings** in the top-right corner of the page, open the **Alert Settings** tab, enable **Email Alerts** related to task validation, and **select recipients**. This ensures timely notifications when data inconsistencies are detected. @@ -60,4 +64,4 @@ To address these challenges, TapData introduces **Incremental Data Validation**, :::tip When differences are found, click “Generate Custom Repair SQL” in the upper-right corner to export the diff SQL for DBA review and targeted fixes. - ::: \ No newline at end of file + ::: diff --git a/docs/data-transformation/process-node.md b/docs/data-transformation/process-node.md index 5e44df07..e305da3c 100644 --- a/docs/data-transformation/process-node.md +++ b/docs/data-transformation/process-node.md @@ -91,6 +91,11 @@ In big data processing and analysis, merging and transforming data is a pivotal Once the process is completed, you can monitor the task's performance on the current page, such as RPS (Records Per Second), latency, task time statistics, etc. + +In addition, when you detect data inconsistencies or unexpected results, stop and edit the task. Click the Master Slave Merge node and, under Cache Management, rebuild the child-table cache that needs to be re-synchronized. + +![Rebuild Data Cache](../images/rebuild_data_cache.png) + ## Union The **Union** node in TapData merges multiple tables with the same or similar structure into a single table, combining the data based on matching field names. The detail rules are as follows: @@ -378,4 +383,3 @@ Parameter Description * [Standard JS](../appendix/standard-js.md): TapData supports processing and operating on data records, providing various functions and operations to manipulate and transform data. For example, you can use JavaScript or Java code to convert date strings to Date types. This allows you to perform date-related operations, comparisons, and formatting on the data records as needed. With this capability, you have flexibility in manipulating and transforming your data to meet your specific requirements. * [Enhanced JS (Beta)](../appendix/enhanced-js.md): TapData supports making external calls in JavaScript code using standard built-in functions. This allows you to perform network requests, interact with databases, and perform other operations by utilizing the capabilities of JavaScript and its built-in functions. - diff --git a/docs/images/audit_api_2.png b/docs/images/audit_api_2.png index 5d3f71f3..542080dd 100644 Binary files a/docs/images/audit_api_2.png and b/docs/images/audit_api_2.png differ diff --git a/docs/images/set_incremental_check.png b/docs/images/set_incremental_check.png index df2815a3..8aabb91f 100644 Binary files a/docs/images/set_incremental_check.png and b/docs/images/set_incremental_check.png differ diff --git a/docs/publish-apis/audit-api.md b/docs/publish-apis/audit-api.md index 91e3522f..a340d83a 100644 --- a/docs/publish-apis/audit-api.md +++ b/docs/publish-apis/audit-api.md @@ -11,8 +11,9 @@ Click **Details** to view detailed information about the corresponding request, ![](../images/audit_api_2.png) -- **Log Details**: Includes basic information and various metrics of the access, such as API ID, name, IP address of the visitor, etc. -- **Number of Access Records**: The total number of records for this access (entries). -- **API Average Access Rate**: The average data transmission speed for a single access to the API. -- **Access Time**: Statistical computation of the time consumed for a single access to the API. -- **Average Response Time**: The average response time per record for a single access to the API. \ No newline at end of file +- **Log details**: Includes basic request info and metrics, such as API ID, name, HTTP method and path, requester IP address, status code, timestamp, etc. +- **Rows returned**: Total number of records returned in this request (independent of pagination or streaming). +- **Matching rows**: Number of records that matched the query/filter conditions (typically ≤ rows returned). +- **API access rate**: Average data transfer rate for a single request, used to evaluate throughput. +- **Request duration**: End-to-end time from receiving the request to completing the response. +- **Database response time**: Time the database took to respond during the request. diff --git a/docs/release-notes-on-prem.md b/docs/release-notes-on-prem.md index 8075dfe0..40a6268f 100644 --- a/docs/release-notes-on-prem.md +++ b/docs/release-notes-on-prem.md @@ -14,6 +14,30 @@ import TabItem from '@theme/TabItem'; ``` + +## 4.11.0 + +### New Features + +- Released [Db2 for i connector](connectors/on-prem-databases/db2-for-i.md) with real-time sync and write support for enterprise data integration. +- [Incremental data validation](data-replication/incremental-check.md) now supports bulk manual verification and repair, configurable validation scope, and per-table ignore lists. +- The [master-child merge node](data-transformation/process-node.md#master-slave-merge) adds child-table cache rebuild to quickly fix cache inconsistencies and protect data quality. +- [Data replication tasks](data-replication/create-task.md) extend exactly-once writes with MongoDB and Sybase as supported targets, ensuring idempotent and consistent writes. +- [Task logs](data-replication/monitor-task.md#-task-log-display-area) add DDL output for easier troubleshooting and audit reviews. +- Incremental reads are now auto-tuned by adjusting batch size to balance throughput and stability. + + +### Enhancements + +- [API audit logs](./publish-apis/audit-api.md) now include database response time. +- Improved API performance for faster overall request handling. +- Improved API debug page results display for better readability and debugging. +- Improved the logic for switching master-child merge tasks into incremental mode. + +### Bug Fixes + +- Fixed a memory leak that could occur in specific scenarios. + ## 4.10.0 ### New Features @@ -985,4 +1009,4 @@ import TabItem from '@theme/TabItem'; - Implementation of error codes for Oracle data sources. - \ No newline at end of file +