Best Practices
- To build efficient and reliable data replication tasks, it is recommended to read the Data Synchronization Best Practices before starting to configure tasks. + To build efficient and reliable data replication tasks, it is recommended to read the Data Synchronization Best Practices before starting to configure tasks.Ways to Add Related Fields
+ +When designing your Incremental Materialized View, you can choose how data from related tables is included in your main record. TapData lets you customize this structure to match your analysis needs and downstream use cases: + +- **Flatten**: Pull selected columns directly into the top level of the main table. Ideal for simple attributes you want to filter or group by (e.g., user_level, country). +- **Embedded Document**: Include all or selected fields as a nested object. Useful for preserving detailed context, such as a user profile with signup date, tier history, or calculated metrics. +- **Embedded Array**: Aggregate multiple related records as an array of objects. Perfect for one-to-many relationships like order items, each enriched with product details. + +By combining these methods, you can design a single view that is analysis-ready, API-friendly, and tailored to your business questions—all without complex joins or heavy ETL processes. + +Understanding type and path in .lookup()
+ These parameters control **how** related data is merged: + + - **type="object"** – embeds the joined record as a nested document at `path`. Ideal for one-to-one enrichments like adding user profiles inside orders. + - **type="array"** – collects multiple matching records as an array of documents at `path`. Perfect for one-to-many relationships, such as order items. + + :::tip + + To flatten joined fields directly into the parent document—so they appear at the same level without any nesting—use `type="object"` and simply leave `path` empty (`path=""`). This creates a wide-table schema that merges all fields into the root level. + + ::: + +What is the role of Agent?
The TapData Agent plays a crucial role in data synchronization, handling data heterogeneity, and supporting data transformation scenarios. It is responsible for extracting data from the source system, performing necessary processing, and transmitting it to the target system. The TapData Agent is centrally managed by TapData Cloud.(Confluent, infra, ops) | + + + +## What Makes TapData Different + +### Built for Speed, Not Complexity + +Launch pipelines across 100+ sources in minutes. No need to wire together Kafka + Debezium + Flink—TapData does it all in one place. + +### Transformations Made Easy + +Join, clean, deduplicate, mask, and enrich data—**visually** or with lightweight JS/SQL. Ideal for both operational and analytical use cases. + +### Incremental Materialized Views (IMV) + +Skip nightly rebuilds. TapData lets you cache joined, aggregated, or filtered views that auto-refresh as source data changes. + +### API-Ready by Design + +Publish any pipeline output as a versioned REST or GraphQL API, complete with row/column-level permissions. No extra backend needed. + +### Layered Architecture + +TapData promotes reusability and control through layered design: + +- **FDM**: Mirror raw source tables via CDC +- **MDM**: Transform into wide, analytics-ready business entities +- **ADM**: Deliver via APIs, pipelines, or data sync + + + +## When TapData Makes Sense + +Use TapData when: + +- You need real-time pipelines **ready in days**, not weeks +- You want **CDC + transform + APIs in one UI** +- You support **business apps** that rely on fresh, accurate data +- You want **low maintenance**—no Kafka tuning, no Flink jobs + +**Example:** + “Sync Salesforce, PostgreSQL, and MongoDB into a real-time user view, apply masking, and publish as an API—in 2 hours.” + + + +## When Kafka Excels + +Kafka is your best bet when: + +- You’re building **custom event architectures** +- You need **ultra-high throughput** (1M+ events/sec) +- Your team has **deep streaming expertise** + +**Example:** + “Build a fraud detection engine with Flink jobs on raw Kafka streams.” + + + +## TapData + Kafka: Better Together + +TapData can simplify and supercharge your Kafka stack. + +| Use Case | Flow Diagram | TapData Benefits | +| --------------------------------- | ------------------------------------------ | ------------------------------------------------------------ | +| **CDC Frontend for Kafka** | DBs → TapData → Kafka Topics | Visual UI for schema handling, masking, filtering, and deduplication | +| **Serving Layer on Top of Kafka** | Kafka Topics → TapData → REST/GraphQL APIs | Exposes topics as secure, queryable APIs—no need to build API layers manually | + +**Why this hybrid works:** +Kafka excels at raw event distribution. TapData brings **developer experience, governance, and business access**—all while staying real-time. + + + +## Final Takeaways + +1. **TapData = Real-Time Simplicity** + Pipelines, transformations, IMVs, and APIs—fully managed, no code required. +2. **Kafka = Custom-Built Power** + Ideal for teams building large-scale, event-first architectures from the ground up. +3. **Together = The Best of Both Worlds** + TapData reduces the engineering effort to adopt, extend, and operationalize Kafka. \ No newline at end of file diff --git a/docs/introduction/terms.md b/docs/introduction/terms.md index 3f40c35d..e0e3bbc5 100644 --- a/docs/introduction/terms.md +++ b/docs/introduction/terms.md @@ -1,62 +1,79 @@ # Terminology -import Content from '../reuse-content/_all-features.md'; +This article introduces common terms used in TapData to help you quickly understand product and feature concepts. -
icon to search, then drag the desired table(s) to the **Platform Cache Layer**.
+
+ 
+
+ :::tip Alternative Approach
+
+ You can also replicate data using the traditional [Data Replication](../../data-replication/create-task.md) approach, which provides more granular control over source-to-target configurations and processing nodes. However, the FDM layer approach shown here is specifically optimized for building an operational data hub with standardized naming and governance.
+
+ :::
+
+5. In the configuration dialog, set a **table prefix** and choose the replication mode (**Full and Incremental Sync** or **Full Sync**).
+
+ 
+
+ Tapdata automatically generates the FDM-layer naming pattern for you, which typically includes the `FDM_` prefix plus your table name.
+ Here you can add your **source system identifier** to make it easier for business teams to recognize the origin of the data.
+
+ For more naming best practices, see [Plan Your Operational Data Hub](../plan-data-platform.md).
+
+6. Click **Save & Run** to immediately start the replication.
+
+ Tapdata will automatically create and launch a real-time replication task, continuously syncing your selected tables into the **FDM (Platform Cache) Layer** with built-in validation.
+
+ To monitor the task, click the 
icon next to the table name—this opens the job monitoring page with live status and performance metrics.
+
+ 
+
+ :::tip
+
+ When multiple tables are selected from the same database source, Tapdata will group them into a single replication task by default. This makes it easier to manage schema changes and monitor task status consistently.
+
+ :::
+
+ Tapdata also helps keep things organized by automatically creating a **folder** in the Platform Cache named after your source connection. Your new replication task will appear inside this folder, making it easier to manage and find related tables from the same source.
+
+ If you want to rename the folder, simply **hover over it**, click the vertical three-dot (**⋮**) icon, and select **Edit** to change its name.
+
+
+
+
+:::tip
+
+**Need more control?**
+If you want to adjust advanced options—like read concurrency, batch size, hash-based sharding (for splitting large tables during full sync), or index replication—click **Only Save** instead of **Save & Run**.
+After saving, locate your new task in the list, click its name, and configure these settings before running it.
+
+:::
+
+## Next Step
+
+- [Explore FDM Table Details](explore-fdm-tables.md)
+- [Data Validation](validate-data-quality.md)
\ No newline at end of file
diff --git a/docs/operational-data-hub/fdm-layer/validate-data-quality.md b/docs/operational-data-hub/fdm-layer/validate-data-quality.md
new file mode 100644
index 00000000..90752acf
--- /dev/null
+++ b/docs/operational-data-hub/fdm-layer/validate-data-quality.md
@@ -0,0 +1,71 @@
+# Validate Data Quality
+
+Tapdata’s data validation feature helps you ensure data integrity throughout your real-time pipelines. Built on robust in-house technology, it verifies that your replicated data remains consistent between source and target, helping you meet strict production-quality requirements. This guide explains how to configure and manage data validation tasks.
+
+## Background
+
+Maintaining trust in your data is critical—especially when delivering real-time insights or meeting strict compliance requirements. Tapdata’s CDC-powered pipelines deliver high-quality replication to the Platform Cache (FDM Layer), while built-in validation tools ensure your data remains consistent, reliable, and fully traceable at every step.
+
+Tapdata provides a flexible data validation framework that helps you:
+
+- Confidently verify record counts and field-level accuracy between your source systems and the Platform Cache (FDM Layer)
+- Quickly detect and analyze discrepancies to support real-time operations
+- Configure alerts and outputs to fit your team’s workflow
+
+Whether you’re managing a single replication job or hundreds of pipelines across teams, Tapdata’s validation features give you complete confidence in your data integrity.
+
+## Limitation
+
+Data validation is **not supported** for queue-based sources like Kafka.
+
+## Configure a Validation Task
+
+1. Log in to TapData Platform.
+
+2. In the left sidebar, select **Data Validation**.
+
+3. On the validation page, click the button in the top-right corner to choose your validation type:
+
+ - **Task Consistency Validation**: Use this to validate data consistency for a specific replication task between your source and FDM layer.
+ - **Any Table Data Validation**: Use this option to validate data between any two tables you have access to in Tapdata, even if they aren’t linked to a specific replication task. This is useful for ad hoc checks or legacy system comparisons.
+
+4. In the setup page, complete the following fields:
+
+ 
+
+ - **Choose Task**: Select the replication task you want to validate. Tasks created when syncing to the FDM layer follow the naming pattern `TableName_Clone_To_FDM_Recommended best practices
+ + +- You can use one shared database for both FDM and MDM layers, or create dedicated databases for better isolation and scalability. +- Deploy MongoDB as a [replica set](../platform-ops/production-deploy/install-replica-mongodb.md) to ensure high availability and fault tolerance. +- Ensure the MongoDB instance has enough disk space and a 14-day Oplog retention to support stable real-time synchronization and CDC. + +Agent support multi-platform installation, see Install Agent.
+Agent support multi-platform installation, see Install Agent.
Click Stop to pause the Agent, which can be used for temporary maintenance scenarios, to restart the Agent later, you should run it from the command line.
diff --git a/docs/billing/refund.md b/docs/other/refund.md similarity index 93% rename from docs/billing/refund.md rename to docs/other/refund.md index fba014e1..55ee587d 100644 --- a/docs/billing/refund.md +++ b/docs/other/refund.md @@ -1,14 +1,12 @@ # Unsubscribe Instance -import Content from '../reuse-content/_cloud-features.md'; -| Category | +Business Load | +CPU Cores Required | +Memory Required | +Number of Pipelines per 16-core Server | +
|---|---|---|---|---|
| Full Synchronization | +Large Data Volume (Table Data > 1 TB) | +4 | +1 GB per 1KB row size | +8 | +
| Medium/Small Data Volume (Table Data < 1TB) | +2 | +16 | +||
| Incremental Replication | +High Throughput (RPS > 10,000) | +2 | +8 | +|
| Medium Throughput (1,000 ~ 9,999 RPS) | +1 | +16 | +||
| Low Throughput (RPS < 1,000) | +0.5 | +32 | +
| Cloud Provider | -Region | -Server Address of Agent | -
|---|---|---|
| Google Cloud | -HongKong | -34.92.78.86 | -
| Sydney | -34.87.244.166 | -|
| Singapore | -35.240.192.89 | -|
| TaiWan | -35.221.187.67 | -|
| Tokyo | -34.146.223.25 | -|
| London | -35.246.16.216 | -|
| Frankfurt | -34.159.220.196 | -|
| N. Virgina | -34.145.229.212 | -|
| Oregon | -34.83.4.199 | -|
| Alibaba Cloud | -HongKong | -47.242.251.110 | -
-
-
-1. [Log in to TapData platform](../log-in.md).
-
-2. In the left navigation bar, click **Data Replication**.
-
-3. Click **Quickly Create Task** on the right side of the page to go to the task creation form.
-
-4. First, select the data source as the source database. You can choose to **Create New Connection** or **Select Existing Connection**.
-
- 
-
- :::tip
-
- The following steps demonstrate how to synchronize MySQL to MongoDB in real-time using the **Select Existing Connection** option. The process is similar for other data sources. For information on how to create data sources in advance, see [Connecting Data Sources](../../prerequisites/README.md).
-
- :::
-
-5. Select **Existing Connection** and choose MySQL as the source database, then click **Next**.
-
-6. Select **Existing Connection** and choose MongoDB as the target database, then click **Next**.
-
-7. In the **Configure Task** step, configure the task details according to the instructions below.
-
- 
-
- * **Task Name**: Enter a meaningful name.
- * **Sync Type**: The default is **Full and Incremental Sync**, but you can also choose **Full Sync** or **Incremental Sync** separately.
- - **Full Sync**: Copies the existing data from the source to the target.
- - **Incremental Sync**: Copies new data or data changes in real-time from the source to the target.
- - The combination of both can be used for real-time data synchronization scenarios.
- * **Duplicate Handling Policy**: Choose based on your business needs. The default is to **Preserve the Original Table Structure and Data on the Target Side**.
- * **Select Tables**: Choose based on your business needs.
- - **Select by Table Name**: Select the tables in the area to be replicated, then click the right arrow to complete the setup.
- - **Select by Regular Expression**: Enter the table name's regular expression. Additionally, if a new table in the source database matches the expression, it will be automatically synchronized to the target database.
- * **Selectable table range**: By default, all tables are displayed, but you can choose to filter only tables with primary keys or only tables without primary keys. Since tables without primary keys use the full primary key method to implement data updates, they might encounter errors due to exceeding the index length limit, and their performance might be limited. Therefore, it is recommended that you create separate data replication tasks for tables without primary keys to avoid task errors and enhance the performance of data updates.
-
-8. Click **Next** to configure more task settings.
-
- 
-
- * **Basic Settings**
- - **Full Multi-thread Writing**: The number of concurrent threads for writing full data. It is disabled by default.
- - **Multi-threaded Incremental Write**: The number of concurrent threads for incremental data writing. It is disabled by default. You can enable and adjust based on the write performance of the target database.
- - **Number of Writes Per Batch**: The number of entries per batch during full synchronization.
- - **Write The Maximum Waiting Time for Each Batch**: Set the maximum wait time based on the target database performance and network latency. The unit is milliseconds.
- - **DDL Synchronization Settings**: Select the DDL event handling strategy. The default is **Automatically Ignore All DDL**. You can choose **Sync DDL Events** and select the DDL events to capture, typically including **Add Column**, **Change Column Name**, **Modify Column Attribute**, **Drop Column**. For more information, see [Handling Schema Changes](../../case-practices/best-practice/handle-schema-changes.md).
- - **Data Read Settings**: Choose the number of entries to read per batch in the full and incremental stages. The default values are 500 and 1, respectively. You can also choose to enable **Enable Concurrent Table Reading** (suitable for scenarios with many small tables).
- - **Data Write Settings**: Choose the data write strategy:
- - **Process by by Event Type**: After selecting this option, you need to choose the data write strategy for insert, update, and delete events.
- - **Append Write**: Only handles insert events, discards update and delete events.
- * **Advanced Settings**
- - **When the Event Processing is Abnormal**: The default is to retry, but you can choose to skip the erroneous event and continue the task.
- - **Other Settings**: Set the task start time, shared mining, periodic scheduling, dynamic memory adjustment, incremental data processing mode, processor threads, Agent, etc.
- * **Alert Settings**: By default, if the average processing time of the node is greater than or equal to 5 seconds for 1 minute, a system and email notification will be sent. You can adjust the rules or disable alerts based on your business needs.
-
-9. After configuration, click **Start Task**.
-
-10. After the task starts successfully, it will automatically jump to the task monitoring page, where you can view information such as RPS (records per second), latency, and task events.
-
- Additionally, to ensure the normal operation of the task, TapData will perform a pre-check based on node configuration and data source characteristics. You can view the printed log information at the bottom of the page.
-
- 
-
-## See also
-
-* [Processing Nodes](process-node.md): Combine multiple processing nodes and data sources to achieve more complex and customized data flow capabilities.
-* [FAQ](../../faq/data-pipeline.md): Common issues and solutions when using the data replication feature.
diff --git a/docs/user-guide/data-development/README.md b/docs/user-guide/data-development/README.md
deleted file mode 100644
index 8290de95..00000000
--- a/docs/user-guide/data-development/README.md
+++ /dev/null
@@ -1,11 +0,0 @@
-# Data Transformation
-
-import Content from '../../reuse-content/_all-features.md';
-
-Best Practices
- To build efficient and reliable data replication tasks, it is recommended to read Data Synchronization Best Practices before starting to configure tasks. - icon to find the table you want to synchronize and drag it to the **Foundation Data Model**.
-
-2. In the pop-up dialog box, fill in the table prefix that is meaningful for your business, select the synchronization type, and choose whether to run the task.
-
- In this case, the table we want to synchronize is the customer, fill in the prefix here is FDM_demo, then in the Master Data Model, the table is called FDM__demo_customer.
-
- 
-
- * **Only Save**: Save the task without running it. You can now click on the task name in the Foundation Data Model to customize the task further.
- * **Save and Run**: No additional action is required. TapData will automatically create a data replication task to synchronize your selected tables in real-time to the Master Data Model and automatically verify. You can click the icon on the right side of the table name in the Foundation Data Model and jump to the task monitoring page to see the task operation details.
-
-
-
- icon to find the table you want to process and drag it to the **Master Data Model**.
-
-2. In the pop-up dialog, fill in the table name and select whether to start the task.
-
- 
-
- * Only Save: Save the task without running it. You can now click on the task name in the target data card to customize the task further. On the redirected task configuration page, you can processing nodes to meet requirements such as table structure adjustment (e.g., adding fields), table merging, and building wide tables. Once the setup is complete, click Start in the upper right corner of the page.
- * **Save and Run**: No further action is necessary as TapData automatically generates a data transformation task and initiates it to synchronize the table in real-time with the Data Processing Layer.
-
-3. At the Data Processing Layer, find the target table and clicking on the icon as below show will allows you to explore the lineage of the table, revealing the chain of relationships that led to the creation of this data table. This feature assists you in effectively managing your tables.
-
- 
-
- In this case, we constructed a wide table called **MDM_demo_customer** based on the initial **customer** and **lineorder** tables.
-
-
icon allows you to view the data source information in the form of a directory structure. You can navigate through the directory structure to select a specific table.
-
-In the catelog view, if you select a specific table, you can also see table details on the right-hand side of the page. The introduction to each tab is as follows:
-
-
-* **Overview**: Provides essential information about tables, including table size, row count, column types, column descriptions (sourced from comments by default), sample data, and more.
-* **Schema**: Offers in-depth insights into table columns, encompassing details like column types, primary keys, foreign keys, default values, and more.
-* **Tasks**: Displays associated tasks for the table, along with their respective statuses. This tab also enables the creation of new tasks.
-* **Lineage**: Presents data lineage relationships visually through a graph format, aiding in effective data quality management. Clicking on a task node allows direct navigation to the monitoring page of the relevant task.
-
-
icon, in the pop-up dialog, we can add a data source, select a data source will jump to the connection configuration page. For more information, see Connect Data Sources.
- icon allows you to enter a keyword for the table name, enabling you to quickly navigate to the specific table. This feature is also supported in other Layers.
- icon,will display the connection information and associated tasks of the data source on the right side of the page.
-
-
- icon, you can view all related data synchronization tasks for this data source, along with their operational statuses and other relevant information.
- icon to find the table you want to synchronize and drag it to the **Foundation Data Model**.
-
-2. In the pop-up dialog box, fill in the table prefix that is meaningful for your business, select the synchronization type, and choose whether to run the task.
-
- In this case, the table we want to synchronize is the customer, fill in the prefix here is FDM_demo, then in the Master Data Model, the table is called FDM__demo_customer.
-
- 
-
- * **Only Save**: Save the task without running it. You can now click on the task name in the Foundation Data Model to customize the task further.
- * **Save and Run**: No additional action is required. TapData will automatically create a data replication task to synchronize your selected tables in real-time to the Master Data Model and automatically verify. You can click the icon on the right side of the table name in the Foundation Data Model and jump to the task monitoring page to see the task operation details.
-
-
-
- icon to find the table you want to process and drag it to the **Master Data Model**.
-
-2. In the pop-up dialog, fill in the table name and select whether to start the task.
-
- 
-
- * Only Save: Save the task without running it. You can now click on the task name in the target data card to customize the task further. On the redirected task configuration page, you can processing nodes to meet requirements such as table structure adjustment (e.g., adding fields), table merging, and building wide tables. Once the setup is complete, click Start in the upper right corner of the page.
- * **Save and Run**: No further action is necessary as TapData automatically generates a data transformation task and initiates it to synchronize the table in real-time with the Data Processing Layer.
-
-3. At the Data Processing Layer, find the target table and clicking on the icon as below show will allows you to explore the lineage of the table, revealing the chain of relationships that led to the creation of this data table. This feature assists you in effectively managing your tables.
-
- 
-
- In this case, we constructed a wide table called **MDM_demo_customer** based on the initial **customer** and **lineorder** tables.
-
- icon allows you to view the data source information in the form of a directory structure. You can navigate through the directory structure to select a specific table.
-
-In the catelog view, if you select a specific table, you can also see table details on the right-hand side of the page. The introduction to each tab is as follows:
-
-
-* **Overview**: Provides essential information about tables, including table size, row count, column types, column descriptions (sourced from comments by default), sample data, and more.
-* **Schema**: Offers in-depth insights into table columns, encompassing details like column types, primary keys, foreign keys, default values, and more.
-* **Tasks**: Displays associated tasks for the table, along with their respective statuses. This tab also enables the creation of new tasks.
-* **Lineage**: Presents data lineage relationships visually through a graph format, aiding in effective data quality management. Clicking on a task node allows direct navigation to the monitoring page of the relevant task.
- 
-
- icon, in the pop-up dialog, we can add a data source, select a data source will jump to the connection configuration page. For more information, see Connect Data Sources.
- icon allows you to enter a keyword for the table name, enabling you to quickly navigate to the specific table. This feature is also supported in other Layers.
- icon,will display the connection information and associated tasks of the data source on the right side of the page.
-
-
- icon, you can view all related data synchronization tasks for this data source, along with their operational statuses and other relevant information.
-Specifications Description
--
-
- M10: 2 vCPUs, 2 GB RAM -
- M20: 2 vCPUs, 4 GB RAM -
- M30: 2 vCPUs, 8 GB RAM -
- M40: 4 vCPUs, 16 GB RAM -
- M50: 8 vCPUs, 32 GB RAM -
- M60: 16 vCPUs, 64 GB RAM -
Want to use an existing MongoDB Atlas?
- At the top of the page, click on click here to privede the connection information, and fill in the MongoDB Atlas connection URL. -Click the icon to display the data source information in the form of a directory structure (click again to switch back to the Console view). Select the specific table, or view the tasks associated with the table and the basic information of the table, including table size, number of rows, column information, sample data, Scheme (such as primary key/foreign key), etc.
-
- Clicking the icon opens a dialog where you can add a data source. Selecting a data source will take you to the connection configuration page. For more information, see Connect Data Sources.
Clicking the icon allows you to enter a keyword for the table name, enabling you to quickly navigate to the specific table. This feature is also supported in other Layers.
- On the right side of the data connection, click the icon will display the connection information and associated tasks of the data source on the right side of the page.
- On the right side of the table name, click the icon. On the right side of the page, the basic information of the tasks and tables associated with the table will be displayed, including table size, number of rows, column information, sample data, Scheme (such as primary key/foreign key), etc.
- Click the 
icon to open the pop-up dialog, you can switch to the Data Service Platform Model.
+
+ );
+};
+
+export default TapDataAnimation;
\ No newline at end of file
diff --git a/src/components/Animation/TapDataAnimation.module.css b/src/components/Animation/TapDataAnimation.module.css
new file mode 100644
index 00000000..7767d8d6
--- /dev/null
+++ b/src/components/Animation/TapDataAnimation.module.css
@@ -0,0 +1,409 @@
+/* TapData Animation Styles */
+.container {
+ max-width: 1200px;
+ margin: 20px auto;
+ position: relative;
+ font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
+ background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
+ min-height: 650px;
+ height: auto;
+ padding: 30px;
+ border-radius: 10px;
+ box-shadow: 0 8px 32px rgba(0, 0, 0, 0.1);
+ overflow: hidden;
+}
+
+.demoArea {
+ display: grid;
+ grid-template-columns: 1.2fr 180px 1fr;
+ gap: 30px;
+ align-items: center;
+ position: relative;
+ max-width: 1200px;
+ margin: 0 auto;
+ min-height: 550px;
+ height: auto;
+}
+
+.sourceSection {
+ display: grid;
+ grid-template-columns: 1fr 1fr;
+ grid-template-rows: 1fr 1fr;
+ gap: 20px;
+}
+
+.processingSection {
+ display: flex;
+ justify-content: center;
+ align-items: center;
+ height: 100%;
+ position: relative;
+}
+
+.resultSection {
+ display: flex;
+ flex-direction: column;
+}
+
+.table {
+ background: white;
+ border-radius: 12px;
+ box-shadow: 0 4px 16px rgba(0,0,0,0.08);
+ overflow: hidden;
+ transition: transform 0.3s ease, box-shadow 0.3s ease;
+ border: 1px solid #e8eaed;
+}
+
+.table:hover {
+ transform: translateY(-3px);
+ box-shadow: 0 8px 24px rgba(0,0,0,0.12);
+}
+
+.tableHeader {
+ background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+ color: white;
+ padding: 12px 15px;
+ font-weight: bold;
+ font-size: 14px;
+ text-align: center;
+}
+
+.field {
+ padding: 10px 15px;
+ border-bottom: 1px solid #f0f0f0;
+ display: flex;
+ justify-content: flex-start;
+ align-items: center;
+ transition: all 0.3s ease;
+ cursor: pointer;
+ font-size: 14px;
+}
+
+.field:last-child {
+ border-bottom: none;
+}
+
+.field:hover {
+ background: #f8f9ff;
+}
+
+.fieldName {
+ font-weight: 500;
+ color: #2c3e50;
+ width: 100%;
+}
+
+.fieldType {
+ font-size: 11px;
+ color: #7f8c8d;
+ background: #ecf0f1;
+ padding: 2px 6px;
+ border-radius: 3px;
+ margin-left: 8px;
+}
+
+.joinKey {
+ background: linear-gradient(90deg, #ffeaa7 0%, #fab1a0 100%);
+ font-weight: bold;
+}
+
+.processingEngine {
+ background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+ border-radius: 18px;
+ padding: 18px;
+ text-align: center;
+ color: white;
+ box-shadow: 0 8px 32px rgba(102, 126, 234, 0.25);
+ position: relative;
+ overflow: hidden;
+ width: 140px;
+ height: 110px;
+ display: flex;
+ flex-direction: column;
+ justify-content: center;
+ z-index: 5;
+}
+
+.engineLogo {
+ font-size: 20px;
+ font-weight: bold;
+ margin-bottom: 8px;
+}
+
+.engineSubtitle {
+ font-size: 12px;
+ opacity: 0.9;
+}
+
+.materializedView {
+ background: white;
+ border-radius: 16px;
+ box-shadow: 0 6px 24px rgba(0,0,0,0.08);
+ overflow: hidden;
+ border: 1px solid #e8eaed;
+}
+
+.mvHeader {
+ background: linear-gradient(135deg, #00b894 0%, #00cec9 100%);
+ color: white;
+ padding: 15px;
+ font-weight: bold;
+ font-size: 16px;
+ text-align: center;
+}
+
+.mvContent {
+ padding: 20px;
+ font-family: 'Courier New', monospace;
+ font-size: 13px;
+ line-height: 1.6;
+ color: #2c3e50;
+ max-height: 600px;
+ overflow-y: auto;
+ transition: background-color 0.3s ease;
+ white-space: pre-wrap;
+}
+
+.mvStructure {
+ position: relative;
+}
+
+.mvLine {
+ margin: 2px 0;
+ opacity: 0;
+ transform: translateX(-10px);
+ transition: all 0.3s ease;
+}
+
+.mvLine.show {
+ opacity: 1;
+ transform: translateX(0);
+}
+
+.mvLine.highlight {
+ background: rgba(102, 126, 234, 0.1);
+ padding: 2px 4px;
+ border-radius: 3px;
+}
+
+/* New field highlight animation - SaaS style light purple */
+:global(.new-field) {
+ animation: newFieldHighlight 1.5s ease-in-out forwards;
+ background: linear-gradient(90deg, #e8e5ff, #d4c5ff);
+ padding: 2px 6px;
+ border-radius: 4px;
+ font-weight: 500;
+ border: 1px solid rgba(116, 75, 162, 0.2);
+ transition: all 0.5s ease-out;
+}
+
+:global(.new-field.fade-out) {
+ background: transparent !important;
+ border-color: transparent !important;
+ box-shadow: none !important;
+ transform: scale(1) !important;
+ padding: 0 !important;
+ font-weight: normal !important;
+}
+
+@keyframes newFieldHighlight {
+ 0% {
+ background: linear-gradient(90deg, #e8e5ff, #d4c5ff);
+ transform: scale(1.03);
+ box-shadow: 0 0 12px rgba(116, 75, 162, 0.4);
+ border-color: rgba(116, 75, 162, 0.4);
+ }
+ 30% {
+ background: linear-gradient(90deg, #d4c5ff, #c8b5ff);
+ transform: scale(1.01);
+ box-shadow: 0 0 16px rgba(116, 75, 162, 0.3);
+ border-color: rgba(116, 75, 162, 0.3);
+ }
+ 60% {
+ background: linear-gradient(90deg, #e8e5ff, #d4c5ff);
+ transform: scale(1.03);
+ box-shadow: 0 0 12px rgba(116, 75, 162, 0.4);
+ border-color: rgba(116, 75, 162, 0.4);
+ }
+ 100% {
+ background: linear-gradient(90deg, #e8e5ff, #d4c5ff);
+ transform: scale(1);
+ box-shadow: 0 0 8px rgba(116, 75, 162, 0.2);
+ border-color: rgba(116, 75, 162, 0.2);
+ }
+}
+
+/* Materialized view content update animation */
+:global(.mv-content.updating) {
+ animation: contentUpdate 0.8s ease-in-out;
+}
+
+@keyframes contentUpdate {
+ 0% {
+ transform: scale(1);
+ box-shadow: 0 4px 20px rgba(0,0,0,0.1);
+ }
+ 50% {
+ transform: scale(1.02);
+ box-shadow: 0 8px 30px rgba(102, 126, 234, 0.3);
+ background: rgba(102, 126, 234, 0.05);
+ }
+ 100% {
+ transform: scale(1);
+ box-shadow: 0 4px 20px rgba(0,0,0,0.1);
+ background: transparent;
+ }
+}
+
+/* Field line by line display animation */
+:global(.field-line) {
+ opacity: 0;
+ transform: translateX(-20px);
+ animation: slideInField 0.5s ease-out forwards;
+}
+
+@keyframes slideInField {
+ 0% {
+ opacity: 0;
+ transform: translateX(-20px);
+ }
+ 100% {
+ opacity: 1;
+ transform: translateX(0);
+ }
+}
+
+:global(.flying-field) {
+ position: absolute;
+ background: rgba(102, 126, 234, 0.9);
+ color: white;
+ padding: 8px 16px;
+ border-radius: 8px;
+ font-size: 16px;
+ font-weight: bold;
+ z-index: 20;
+ pointer-events: none;
+ white-space: nowrap;
+ box-shadow: 0 4px 12px rgba(0, 0, 0, 0.2);
+ opacity: 0;
+ transform: translateX(-50px);
+ min-width: 80px;
+ text-align: center;
+}
+
+:global(.flying-field.animate) {
+ animation: flyInFromLeft 1.2s cubic-bezier(0.25, 0.46, 0.45, 0.94) forwards;
+}
+
+@keyframes flyInFromLeft {
+ 0% {
+ opacity: 0;
+ transform: translateX(-100px) scale(0.5);
+ }
+ 30% {
+ opacity: 1;
+ transform: translateX(-20px) scale(0.8);
+ }
+ 70% {
+ opacity: 1;
+ transform: translateX(20px) scale(1.1);
+ }
+ 100% {
+ opacity: 0;
+ transform: translateX(100px) scale(0.5);
+ }
+}
+
+:global(.field.highlight) {
+ background: rgba(102, 126, 234, 0.2);
+ transform: scale(1.02);
+ box-shadow: 0 0 10px rgba(102, 126, 234, 0.3);
+}
+
+:global(.pulse) {
+ animation: pulse 2s infinite;
+}
+
+@keyframes pulse {
+ 0%, 100% {
+ transform: scale(1);
+ box-shadow: 0 8px 25px rgba(102, 126, 234, 0.3);
+ }
+ 50% {
+ transform: scale(1.05);
+ box-shadow: 0 12px 35px rgba(102, 126, 234, 0.5);
+ }
+}
+
+:global(.engine-processing) {
+ animation: engineJiggle 0.8s ease-in-out infinite;
+}
+
+@keyframes engineJiggle {
+ 0%, 100% {
+ transform: translateX(0) translateY(0);
+ }
+ 25% {
+ transform: translateX(-1px) translateY(-1px);
+ }
+ 50% {
+ transform: translateX(1px) translateY(-1px);
+ }
+ 75% {
+ transform: translateX(-1px) translateY(1px);
+ }
+}
+
+:global(.flow-arrows) {
+ position: absolute;
+ display: flex;
+ align-items: center;
+ gap: 15px;
+ z-index: 10;
+}
+
+:global(.arrow) {
+ font-size: 20px;
+ color: #667eea;
+ opacity: 0;
+ animation: arrowFlow 2s ease-in-out infinite;
+}
+
+:global(.arrow:nth-child(1)) {
+ animation-delay: 0s;
+}
+
+:global(.arrow:nth-child(2)) {
+ animation-delay: 0.3s;
+}
+
+:global(.arrow:nth-child(3)) {
+ animation-delay: 0.6s;
+}
+
+@keyframes arrowFlow {
+ 0%, 70% {
+ opacity: 0;
+ transform: scale(0.8);
+ }
+ 15%, 55% {
+ opacity: 1;
+ transform: scale(1.2);
+ }
+ 30%, 40% {
+ opacity: 0.8;
+ transform: scale(1);
+ }
+}
+
+@media (max-width: 1200px) {
+ .demoArea {
+ grid-template-columns: 1fr;
+ gap: 30px;
+ }
+
+ .sourceSection {
+ grid-template-columns: 1fr;
+ grid-template-rows: auto;
+ }
+}
diff --git a/src/components/Animation/TapDataFlowAnimation.js b/src/components/Animation/TapDataFlowAnimation.js
new file mode 100644
index 00000000..0b62d3b0
--- /dev/null
+++ b/src/components/Animation/TapDataFlowAnimation.js
@@ -0,0 +1,323 @@
+import React, { useEffect, useRef } from 'react';
+import styles from './TapDataFlowAnimation.module.css';
+
+const TapDataFlowAnimation = () => {
+ const containerRef = useRef(null);
+
+ useEffect(() => {
+ if (!containerRef.current) return;
+
+ const container = containerRef.current;
+ const usersTable = container.querySelector('.users');
+ const ordersTable = container.querySelector('.orders');
+ const tapdataPlatform = container.querySelector('.tapdata-platform');
+ const materializedView = container.querySelector('.mv-container');
+ const applications = container.querySelector('.applications');
+
+ const createDataChangeIndicator = (table, type) => {
+ if (!table) return;
+
+ const indicator = document.createElement('div');
+ indicator.className = `data-indicator ${type}`;
+
+ const rect = table.getBoundingClientRect();
+ const containerRect = container.getBoundingClientRect();
+
+ indicator.style.left = (rect.left - containerRect.left + Math.random() * rect.width) + 'px';
+ indicator.style.top = (rect.top - containerRect.top + Math.random() * rect.height) + 'px';
+
+ container.appendChild(indicator);
+
+ setTimeout(() => {
+ if (indicator.parentNode) {
+ indicator.remove();
+ }
+ }, 2000);
+ };
+
+ const createDataFlow = (fromElement, toElement, dataType) => {
+ if (!fromElement || !toElement) return;
+
+ const dataFlow = document.createElement('div');
+ dataFlow.className = `data-flow ${dataType} active`;
+
+ const fromRect = fromElement.getBoundingClientRect();
+ const toRect = toElement.getBoundingClientRect();
+ const containerRect = container.getBoundingClientRect();
+
+ const startX = fromRect.right - containerRect.left;
+ const startY = fromRect.top + fromRect.height / 2 - containerRect.top;
+ const endX = toRect.left - containerRect.left;
+ const endY = toRect.top + toRect.height / 2 - containerRect.top;
+
+ dataFlow.style.left = startX + 'px';
+ dataFlow.style.top = startY + 'px';
+
+ container.appendChild(dataFlow);
+
+ const distance = Math.sqrt(Math.pow(endX - startX, 2) + Math.pow(endY - startY, 2));
+ const duration = Math.max(2000, distance * 5);
+
+ dataFlow.style.transition = `all ${duration}ms ease-in-out`;
+
+ setTimeout(() => {
+ dataFlow.style.left = endX + 'px';
+ dataFlow.style.top = endY + 'px';
+ dataFlow.style.opacity = '0';
+ }, 50);
+
+ setTimeout(() => {
+ if (dataFlow.parentNode) {
+ dataFlow.remove();
+ }
+ }, duration + 100);
+ };
+
+ const startDataChangeIndicators = () => {
+ setInterval(() => {
+ createDataChangeIndicator(usersTable, 'users');
+ }, 800);
+
+ setInterval(() => {
+ createDataChangeIndicator(ordersTable, 'orders');
+ }, 900);
+
+ setInterval(() => {
+ for (let i = 0; i < 5; i++) {
+ setTimeout(() => {
+ createDataChangeIndicator(usersTable, 'users');
+ }, i * 150);
+ }
+ }, 2500);
+
+ setInterval(() => {
+ for (let i = 0; i < 5; i++) {
+ setTimeout(() => {
+ createDataChangeIndicator(ordersTable, 'orders');
+ }, i * 180);
+ }
+ }, 2800);
+ };
+
+ const startDataFlowAnimations = () => {
+ setInterval(() => {
+ createDataFlow(usersTable, tapdataPlatform, 'users-data');
+ }, 1000);
+
+ setInterval(() => {
+ createDataFlow(ordersTable, tapdataPlatform, 'orders-data');
+ }, 1200);
+
+ setInterval(() => {
+ createDataFlow(tapdataPlatform, materializedView, 'merged-data');
+ }, 1400);
+
+ setInterval(() => {
+ createDataFlow(materializedView, applications, 'merged-data');
+ }, 1600);
+
+ setInterval(() => {
+ createDataFlow(usersTable, tapdataPlatform, 'users-data');
+ setTimeout(() => {
+ createDataFlow(ordersTable, tapdataPlatform, 'orders-data');
+ }, 300);
+ }, 2500);
+
+ setInterval(() => {
+ createDataFlow(tapdataPlatform, materializedView, 'merged-data');
+ setTimeout(() => {
+ createDataFlow(materializedView, applications, 'merged-data');
+ }, 400);
+ }, 3000);
+ };
+
+ setTimeout(() => {
+ startDataChangeIndicators();
+ startDataFlowAnimations();
+ }, 3000);
+
+ return () => {
+ const indicators = container.querySelectorAll('.data-indicator');
+ const flows = container.querySelectorAll('.data-flow');
+
+ indicators.forEach(el => el.remove());
+ flows.forEach(el => el.remove());
+ };
+ }, []);
+
+ return (
+
+ {/* 源表部分 */}
+
+
+
+
+ {/* 处理引擎 */}
+
+
+
+ orders
+
+ 🔑 order_id
+
+
+ 🔗 user_id
+
+
+ order_status
+
+
+ order_amount
+
+
+ payment_method
+
+
+ order_time
+
+
+
+
+ users
+
+ 🔑 user_id
+
+
+ user_name
+
+
+ user_level
+
+
+ country
+
+
+ city
+
+
+ signup_time
+
+
+
+
+ order_items
+
+ item_id
+
+
+ 🔗 order_id
+
+
+ 🔗 product_id
+
+
+ quantity
+
+
+
+ products
+
+ 🔑 product_id
+
+
+ product_name
+
+
+ category
+
+
+ unit_price
+
+
+
+
+ {/* 物化视图 */}
+
+
+ TapData
+ Real-time Data
+ Processing Engine
+
+
+
+
+ Materialized View
+ {'{\n\n}'}
+
+
+
+
+ );
+};
+
+export default TapDataFlowAnimation;
diff --git a/src/components/Animation/TapDataFlowAnimation.module.css b/src/components/Animation/TapDataFlowAnimation.module.css
new file mode 100644
index 00000000..1933f390
--- /dev/null
+++ b/src/components/Animation/TapDataFlowAnimation.module.css
@@ -0,0 +1,676 @@
+/* TapData Flow Animation - Adapted for Docusaurus */
+.container {
+ font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
+ background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
+ min-height: 500px;
+ padding: 20px;
+ position: relative;
+ width: 100%;
+ margin: 20px 0;
+ box-sizing: border-box;
+ border-radius: 16px;
+ overflow: hidden;
+}
+
+.title {
+ text-align: center;
+ font-size: 32px;
+ font-weight: 600;
+ color: #2c3e50;
+ margin-bottom: 50px;
+ text-shadow: 0 2px 4px rgba(0,0,0,0.1);
+}
+
+.flowContainer {
+ display: flex;
+ align-items: center;
+ justify-content: space-between;
+ gap: 2%;
+ position: relative;
+ width: 100%;
+ margin: 0 auto;
+ min-height: 400px;
+}
+
+/* Multi-Tables Section */
+.multiTables {
+ flex: 1.5;
+ min-width: 0;
+ opacity: 0;
+ animation: slideInLeft 1s ease-out 0.5s forwards;
+}
+
+.sectionTitle {
+ font-size: 16px;
+ font-weight: 600;
+ color: #4a5568;
+ margin-bottom: 20px;
+ text-align: center;
+ position: relative;
+}
+
+.valueBadge {
+ position: absolute;
+ top: -45px;
+ left: 50%;
+ transform: translateX(-50%);
+ background: linear-gradient(135deg, #ffffff 0%, #f8fafc 100%);
+ border: 2px solid;
+ border-radius: 20px;
+ padding: 6px 14px;
+ font-size: 11px;
+ font-weight: 700;
+ text-transform: uppercase;
+ letter-spacing: 0.5px;
+ box-shadow: 0 4px 12px rgba(0,0,0,0.1);
+ opacity: 0;
+ animation: badgeSlideIn 1s ease-out forwards;
+ z-index: 20;
+ white-space: nowrap;
+}
+
+.valueBadge.zeroImpact {
+ color: #8b5cf6;
+ border-color: #8b5cf6;
+ animation-delay: 4s;
+}
+
+.tablesContainer {
+ display: flex;
+ flex-direction: column;
+ gap: 15px;
+}
+
+.table {
+ background: white;
+ border-radius: 8px;
+ box-shadow: 0 4px 12px rgba(0,0,0,0.1);
+ overflow: hidden;
+ border: 2px solid transparent;
+ transition: all 0.3s ease;
+}
+
+.table.users {
+ border-color: #f59e0b;
+}
+
+.table.orders {
+ border-color: #3182ce;
+}
+
+.tableHeader {
+ padding: 12px 16px;
+ font-weight: 600;
+ font-size: 14px;
+ color: white;
+}
+
+.table.users .tableHeader {
+ background: #f59e0b;
+}
+
+.table.orders .tableHeader {
+ background: #3182ce;
+}
+
+.tableRow {
+ padding: 8px 16px;
+ font-size: 12px;
+ color: #718096;
+ border-bottom: 1px solid #e2e8f0;
+ display: flex;
+ justify-content: space-between;
+}
+
+.tableRow:last-child {
+ border-bottom: none;
+}
+
+/* Processing Section */
+.processing {
+ flex: 2.8;
+ min-width: 0;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ gap: 20px;
+ opacity: 0;
+ animation: fadeIn 1s ease-out 1s forwards;
+}
+
+.tapdataPlatform {
+ background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+ color: white;
+ padding: 20px 24px;
+ border-radius: 12px;
+ text-align: center;
+ box-shadow: 0 8px 20px rgba(102, 126, 234, 0.3);
+ transform: scale(0.8);
+ animation: scaleUp 0.8s ease-out 1.5s forwards, platformPulse 3s ease-in-out infinite 2.5s;
+ position: relative;
+ overflow: visible;
+ width: 100%;
+ max-width: 280px;
+}
+
+/* Data processing rings */
+.tapdataPlatform::before {
+ content: '';
+ position: absolute;
+ top: 50%;
+ left: 50%;
+ width: 120%;
+ height: 120%;
+ border: 2px solid rgba(255, 255, 255, 0.3);
+ border-radius: 50%;
+ transform: translate(-50%, -50%);
+ animation: processingRing1 2s ease-in-out infinite;
+}
+
+.tapdataPlatform::after {
+ content: '';
+ position: absolute;
+ top: 50%;
+ left: 50%;
+ width: 140%;
+ height: 140%;
+ border: 1px solid rgba(255, 255, 255, 0.2);
+ border-radius: 50%;
+ transform: translate(-50%, -50%);
+ animation: processingRing2 2s ease-in-out infinite 0.5s;
+}
+
+.processingIndicator {
+ position: absolute;
+ top: 8px;
+ right: 8px;
+ width: 10px;
+ height: 10px;
+ background: #10b981;
+ border-radius: 50%;
+ animation: processingPulse 1.5s ease-in-out infinite;
+ box-shadow: 0 0 8px rgba(16, 185, 129, 0.6);
+}
+
+.dataProcessor {
+ position: absolute;
+ top: 6px;
+ left: 6px;
+ width: 40px;
+ height: 40px;
+ pointer-events: none;
+ z-index: 10;
+}
+
+.dataProcessor svg {
+ width: 100%;
+ height: 100%;
+ animation: engineJiggle 4s ease-in-out infinite;
+}
+
+.platformTitle {
+ font-size: 18px;
+ font-weight: 600;
+ margin-bottom: 8px;
+}
+
+.platformSubtitle {
+ font-size: 14px;
+ opacity: 0.9;
+}
+
+/* Double Flow Arrows */
+.arrow {
+ width: 60px !important;
+ height: 40px !important;
+ position: relative;
+ opacity: 0;
+ display: flex;
+ align-items: center;
+ justify-content: space-between;
+ margin: 0 auto;
+ flex: 0 0 60px;
+}
+
+.arrow .flowChevron {
+ width: 20px !important;
+ height: 20px !important;
+ opacity: 0;
+ flex-shrink: 0;
+}
+
+.arrow .flowChevron:nth-child(1) {
+ animation: chevronFlow 2s ease-in-out infinite;
+}
+
+.arrow .flowChevron:nth-child(2) {
+ animation: chevronFlow 2s ease-in-out infinite 0.4s;
+}
+
+.arrow .flowChevron svg {
+ width: 100%;
+ height: 100%;
+}
+
+.arrow .flowChevron svg path {
+ fill: #3b82f6;
+}
+
+.arrow.animated {
+ animation: drawArrow 0.8s ease-out forwards;
+}
+
+.arrow.animated .flowChevron:nth-child(1) {
+ animation: drawArrow 0.8s ease-out forwards, chevronFlow 2s ease-in-out infinite 1s;
+}
+
+.arrow.animated .flowChevron:nth-child(2) {
+ animation: drawArrow 0.8s ease-out forwards, chevronFlow 2s ease-in-out infinite 1.4s;
+}
+
+.arrow:nth-of-type(1) { animation-delay: 1.2s; }
+.arrow:nth-of-type(2) { animation-delay: 1.8s; }
+.arrow:nth-of-type(3) { animation-delay: 2.3s; }
+
+/* Materialized View */
+.materializedView {
+ flex: 3.2;
+ min-width: 0;
+ opacity: 0;
+ animation: slideInRight 1s ease-out 2s forwards;
+}
+
+.mvContainer {
+ background: white;
+ border-radius: 16px;
+ box-shadow: 0 6px 16px rgba(56, 161, 105, 0.2);
+ overflow: hidden;
+ border: 2px solid #38a169;
+ width: 100%;
+}
+
+.mvHeader {
+ background: #38a169;
+ color: white;
+ padding: 16px;
+ text-align: center;
+ font-weight: 600;
+ position: relative;
+ padding-right: 65px;
+}
+
+.realtimeIndicator {
+ position: absolute;
+ top: 10px;
+ right: 10px;
+ background: #10b981;
+ color: white;
+ padding: 6px 12px;
+ border-radius: 12px;
+ font-size: 12px;
+ font-weight: 600;
+ animation: blink 2s ease-in-out infinite;
+ z-index: 10;
+ letter-spacing: 0.5px;
+}
+
+.mvContent {
+ padding: 16px;
+ background: #f0fff4;
+}
+
+.mvFeature {
+ display: flex;
+ align-items: center;
+ gap: 8px;
+ margin-bottom: 8px;
+ font-size: 12px;
+ color: #2d3748;
+}
+
+.mvFeature:last-child {
+ margin-bottom: 0;
+}
+
+.checkmark {
+ color: #38a169;
+ font-weight: bold;
+}
+
+/* Applications */
+.applications {
+ flex: 1.5;
+ min-width: 0;
+ opacity: 0;
+ animation: slideInRight 1s ease-out 2.5s forwards;
+}
+
+.appsContainer {
+ display: flex;
+ flex-direction: column;
+ gap: 10px;
+ align-items: center;
+ animation: appsFloat 4s ease-in-out infinite;
+}
+
+.appItem {
+ display: flex;
+ align-items: center;
+ gap: 10px;
+ padding: 10px 14px;
+ background: linear-gradient(135deg, #ffffff 0%, #f8fafc 100%);
+ border-radius: 12px;
+ box-shadow: 0 2px 8px rgba(0,0,0,0.06);
+ font-size: 12px;
+ color: #4b5563;
+ width: 100%;
+ transition: all 0.3s ease;
+ border: 1px solid #e5e7eb;
+ font-weight: 500;
+ opacity: 0;
+ animation: appItemAppear 0.6s ease-out forwards;
+ justify-content: flex-start;
+}
+
+.appItem:nth-child(1) { animation-delay: 3s; }
+.appItem:nth-child(2) { animation-delay: 3.2s; }
+.appItem:nth-child(3) { animation-delay: 3.4s; }
+.appItem:nth-child(4) { animation-delay: 3.6s; }
+.appItem:nth-child(5) { animation-delay: 3.8s; }
+
+.appItem:hover {
+ transform: translateY(-2px);
+ box-shadow: 0 4px 16px rgba(0,0,0,0.12);
+ background: linear-gradient(135deg, #ffffff 0%, #f1f5f9 100%);
+ border-color: #d1d5db;
+}
+
+/* Data Flow Animation - Global styles */
+:global(.data-flow) {
+ position: absolute;
+ width: 10px;
+ height: 10px;
+ border-radius: 50%;
+ opacity: 0;
+ z-index: 10;
+}
+
+:global(.data-flow.users-data) {
+ background: linear-gradient(45deg, #f59e0b, #fbbf24);
+ box-shadow: 0 0 10px rgba(245, 158, 11, 0.5);
+}
+
+:global(.data-flow.orders-data) {
+ background: linear-gradient(45deg, #3182ce, #63b3ed);
+ box-shadow: 0 0 10px rgba(49, 130, 206, 0.5);
+}
+
+:global(.data-flow.merged-data) {
+ background: linear-gradient(45deg, #38a169, #68d391);
+ box-shadow: 0 0 10px rgba(56, 161, 105, 0.5);
+}
+
+:global(.data-flow.active) {
+ animation: flowData 3s ease-in-out infinite;
+}
+
+/* Data change indicators */
+:global(.data-indicator) {
+ position: absolute;
+ width: 10px;
+ height: 10px;
+ background: #f56565;
+ border-radius: 50%;
+ opacity: 0;
+ animation: pulse 2s ease-in-out infinite;
+ box-shadow: 0 0 6px rgba(0,0,0,0.2);
+}
+
+:global(.data-indicator.users) { background: #f59e0b; }
+:global(.data-indicator.orders) { background: #3182ce; }
+
+/* Animations */
+@keyframes fadeIn {
+ from { opacity: 0; }
+ to { opacity: 1; }
+}
+
+@keyframes slideInLeft {
+ from {
+ opacity: 0;
+ transform: translateX(-50px);
+ }
+ to {
+ opacity: 1;
+ transform: translateX(0);
+ }
+}
+
+@keyframes slideInRight {
+ from {
+ opacity: 0;
+ transform: translateX(50px);
+ }
+ to {
+ opacity: 1;
+ transform: translateX(0);
+ }
+}
+
+@keyframes scaleUp {
+ from { transform: scale(0.8); }
+ to { transform: scale(1); }
+}
+
+@keyframes badgeSlideIn {
+ 0% {
+ opacity: 0;
+ transform: translateX(-50%) translateY(20px) scale(0.8);
+ }
+ 100% {
+ opacity: 1;
+ transform: translateX(-50%) translateY(0) scale(1);
+ }
+}
+
+@keyframes platformPulse {
+ 0%, 100% {
+ transform: scale(1);
+ box-shadow: 0 8px 20px rgba(102, 126, 234, 0.3);
+ }
+ 50% {
+ transform: scale(1.05);
+ box-shadow: 0 12px 30px rgba(102, 126, 234, 0.5);
+ }
+}
+
+@keyframes processingRing1 {
+ 0% {
+ transform: translate(-50%, -50%) scale(0.8);
+ opacity: 0.8;
+ }
+ 50% {
+ transform: translate(-50%, -50%) scale(1);
+ opacity: 0.4;
+ }
+ 100% {
+ transform: translate(-50%, -50%) scale(1.2);
+ opacity: 0;
+ }
+}
+
+@keyframes processingRing2 {
+ 0% {
+ transform: translate(-50%, -50%) scale(0.8);
+ opacity: 0.6;
+ }
+ 50% {
+ transform: translate(-50%, -50%) scale(1);
+ opacity: 0.3;
+ }
+ 100% {
+ transform: translate(-50%, -50%) scale(1.2);
+ opacity: 0;
+ }
+}
+
+@keyframes processingPulse {
+ 0%, 100% {
+ opacity: 1;
+ transform: scale(1);
+ box-shadow: 0 0 8px rgba(16, 185, 129, 0.6);
+ }
+ 50% {
+ opacity: 0.7;
+ transform: scale(1.3);
+ box-shadow: 0 0 15px rgba(16, 185, 129, 0.8);
+ }
+}
+
+@keyframes engineJiggle {
+ 0%, 85%, 100% { transform: translateX(0) translateY(0) rotate(0deg); }
+ 1% { transform: translateX(-1px) translateY(0.5px) rotate(-0.5deg); }
+ 2% { transform: translateX(1px) translateY(-0.5px) rotate(0.5deg); }
+ 3% { transform: translateX(-0.5px) translateY(1px) rotate(-0.3deg); }
+ 4% { transform: translateX(0.5px) translateY(-1px) rotate(0.3deg); }
+ 5% { transform: translateX(-1px) translateY(-0.5px) rotate(-0.5deg); }
+ 6% { transform: translateX(1px) translateY(0.5px) rotate(0.5deg); }
+ 7% { transform: translateX(-0.5px) translateY(-1px) rotate(-0.3deg); }
+ 8% { transform: translateX(0.5px) translateY(1px) rotate(0.3deg); }
+ 9% { transform: translateX(-1px) translateY(0) rotate(-0.5deg); }
+ 10% { transform: translateX(0) translateY(0) rotate(0deg); }
+}
+
+@keyframes chevronFlow {
+ 0% {
+ opacity: 0;
+ transform: translateX(-10px) scale(0.8);
+ }
+ 50% {
+ opacity: 1;
+ transform: translateX(0) scale(1);
+ }
+ 100% {
+ opacity: 0;
+ transform: translateX(10px) scale(0.8);
+ }
+}
+
+@keyframes drawArrow {
+ from {
+ opacity: 0;
+ width: 0;
+ }
+ to {
+ opacity: 1;
+ width: 60px;
+ }
+}
+
+@keyframes appsFloat {
+ 0%, 100% { transform: translateY(0); }
+ 50% { transform: translateY(-8px); }
+}
+
+@keyframes appItemAppear {
+ 0% {
+ opacity: 0;
+ transform: translateY(20px);
+ }
+ 100% {
+ opacity: 1;
+ transform: translateY(0);
+ }
+}
+
+@keyframes blink {
+ 0%, 50% { opacity: 1; }
+ 25%, 75% { opacity: 0.5; }
+}
+
+@keyframes flowData {
+ 0% { opacity: 0; transform: translateX(0) scale(0.5); }
+ 20% { opacity: 1; transform: translateX(20%) scale(1); }
+ 80% { opacity: 1; transform: translateX(80%) scale(1); }
+ 100% { opacity: 0; transform: translateX(100%) scale(0.5); }
+}
+
+@keyframes pulse {
+ 0%, 100% { opacity: 0; transform: scale(0.5); }
+ 50% { opacity: 1; transform: scale(1.5); }
+}
+
+/* Responsive Design */
+@media (max-width: 1200px) {
+ .flowContainer {
+ gap: 1.5%;
+ }
+
+ .arrow {
+ width: 50px !important;
+ flex: 0 0 50px;
+ }
+
+ .arrow .flowChevron {
+ width: 16px !important;
+ height: 16px !important;
+ }
+}
+
+@media (max-width: 900px) {
+ .flowContainer {
+ gap: 1%;
+ }
+
+ .arrow {
+ width: 40px !important;
+ flex: 0 0 40px;
+ }
+
+ .arrow .flowChevron {
+ width: 14px !important;
+ height: 14px !important;
+ }
+
+ .appItem {
+ font-size: 11px;
+ padding: 8px 12px;
+ gap: 8px;
+ }
+
+ .tapdataPlatform {
+ max-width: 200px;
+ padding: 16px 20px;
+ }
+}
+
+@media (max-width: 768px) {
+ .container {
+ min-height: 400px;
+ padding: 15px;
+ }
+
+ .flowContainer {
+ flex-direction: column;
+ gap: 20px;
+ }
+
+ .multiTables,
+ .processing,
+ .materializedView,
+ .applications {
+ flex: none;
+ width: 100%;
+ }
+
+ .arrow {
+ transform: rotate(90deg);
+ width: 40px !important;
+ height: 20px !important;
+ }
+
+ .arrow .flowChevron {
+ width: 12px !important;
+ height: 12px !important;
+ }
+}
diff --git a/tools/README.md b/tools/README.md
new file mode 100644
index 00000000..9390b7d9
--- /dev/null
+++ b/tools/README.md
@@ -0,0 +1,295 @@
+# Broken Links Checker for Docusaurus
+
+A comprehensive tool for detecting broken internal links in Docusaurus documentation sites. This tool provides both detailed and quick checking modes with Excel export functionality.
+
+## Features
+
+- 🔍 **Smart Detection**: Identifies broken relative path links, parent directory links, and internal references
+- 📊 **Excel Export**: Generates CSV files compatible with Excel for easy analysis
+- 🚀 **Dual Modes**: Quick check for rapid overview, detailed check for comprehensive analysis
+- 📈 **Statistics**: Provides breakdown by link type and affected files
+- 💡 **Fix Suggestions**: Offers actionable recommendations for resolving broken links
+- 🎯 **Docusaurus Optimized**: Understands Docusaurus file structure and routing conventions
+
+## Installation
+
+No additional dependencies required - uses Node.js built-in modules only.
+
+## Usage
+
+### Quick Check (Recommended for regular use)
+
+```bash
+npm run check-links:quick
+```
+
+**Features:**
+- Fast scanning of all Markdown files
+- Concise table output
+- Basic statistics
+- CSV export for Excel
+
+**Output Example:**
+```
+🚀 Quick broken link check...
+
+📁 Scanning 45 files...
+
+📊 Quick Check Results
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ Broken Links Summary │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ File │ Type │ Broken Link │
+├─────────────────────────────────────────────────────────────────────────────┤
+│ getting-started/installation │ Parent Dir │ ../overview/introduction │
+│ api/authentication.md │ Relative │ ./examples/basic-auth │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+📈 Summary:
+ Total broken links: 39
+ Affected files: 25
+ Parent Dir: 21
+ Relative: 18
+
+💡 Run "npm run check-links" for detailed report with Excel export
+💾 Quick report saved: quick-broken-links-2025-01-26T10-30-45-123Z.csv
+```
+
+### Detailed Check (For comprehensive analysis)
+
+```bash
+npm run check-links
+```
+
+**Features:**
+- Line-by-line analysis
+- Detailed link information
+- Comprehensive Excel reports
+- Fix suggestions
+- JSON export for automation
+
+**Output Example:**
+```
+🔍 Starting broken link detection...
+
+📁 Found 156 documentation files
+
+📊 Broken Link Detection Results
+
+❌ Found 38 broken links:
+
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Broken Links Report │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
+│ File Path │ Line │ Link Type │ Link Text │ Target Path │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
+│ getting-started/install... │ 15 │ Parent Directory │ Overview │ ../overview/intro │
+│ │ 23 │ Relative Path │ Configuration Guide │ ./config/setup │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+
+📈 Statistics:
+ Total broken links: 38
+ Affected files: 23
+ By type:
+ Parent Directory: 22
+ Relative Path: 16
+
+💡 Fix Suggestions:
+ 1. Check if target files exist
+ 2. Verify file paths are correct
+ 3. Check file name case sensitivity
+ 4. Verify relative path hierarchy
+
+💾 Reports saved:
+ Excel/CSV: broken-links-report-2025-01-26T10-30-45-123Z.csv
+ JSON: broken-links-report-2025-01-26T10-30-45-123Z.json
+```
+
+### Direct Script Execution
+
+You can also run the scripts directly:
+
+```bash
+# Quick check
+node tools/quick-check-links.js
+
+# Detailed check
+node tools/check-broken-links.js
+```
+
+## Output Files
+
+### CSV/Excel Reports
+- **Quick Check**: `quick-broken-links-[timestamp].csv`
+- **Detailed Check**: `broken-links-report-[timestamp].csv`
+
+CSV files can be opened directly in Excel and contain:
+- File paths
+- Line numbers (detailed mode only)
+- Link types
+- Link text
+- Target paths
+
+### JSON Reports (Detailed mode only)
+- **Filename**: `broken-links-report-[timestamp].json`
+- **Content**: Machine-readable format for automation and CI/CD integration
+
+## Link Types
+
+The tool categorizes broken links into these types:
+
+- **Parent Directory**: Links starting with `../` (e.g., `../other-section/file.md`)
+- **Current Directory**: Links starting with `./` (e.g., `./local-file.md`)
+- **Root Directory**: Links starting with `/` (e.g., `/docs/api/reference.md`)
+- **Relative Path**: Other relative links (e.g., `subfolder/file.md`)
+
+## Common Broken Link Causes
+
+1. **File Moved/Renamed**: Target file was moved or renamed
+2. **Case Sensitivity**: Incorrect capitalization in file names
+3. **Missing Extension**: Link missing `.md` or `.mdx` extension
+4. **Wrong Path**: Incorrect relative path calculation
+5. **Deleted Files**: Referenced file was deleted
+
+## Fix Recommendations
+
+### For Parent Directory Links (`../`)
+```markdown
+# ❌ Broken
+[Guide](../missing-section/guide.md)
+
+# ✅ Fixed
+[Guide](../existing-section/guide.md)
+```
+
+### For Relative Path Links
+```markdown
+# ❌ Broken
+[API Reference](api/missing-endpoint.md)
+
+# ✅ Fixed
+[API Reference](api/existing-endpoint.md)
+```
+
+### For Case Sensitivity Issues
+```markdown
+# ❌ Broken (wrong case)
+[Setup](./Setup.md)
+
+# ✅ Fixed (correct case)
+[Setup](./setup.md)
+```
+
+## Configuration
+
+### Custom Documentation Directory
+
+Modify the scripts to use a different documentation directory:
+
+```javascript
+// In tools/check-broken-links.js or tools/quick-check-links.js
+const checker = new BrokenLinkChecker('./custom-docs-dir');
+```
+
+### File Extensions
+
+The tool checks these file extensions by default:
+- `.md` (Markdown)
+- `.mdx` (MDX)
+
+To add more extensions, modify the `validExtensions` array in the scripts.
+
+## Integration with CI/CD
+
+### GitHub Actions Example
+
+```yaml
+name: Check Broken Links
+
+on:
+ pull_request:
+ paths:
+ - 'docs/**'
+ - '*.md'
+
+jobs:
+ check-links:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
+ - uses: actions/setup-node@v3
+ with:
+ node-version: '18'
+ - run: npm run check-links:quick
+ - name: Upload broken links report
+ if: failure()
+ uses: actions/upload-artifact@v3
+ with:
+ name: broken-links-report
+ path: quick-broken-links-*.csv
+```
+
+### Pre-commit Hook
+
+Add to `.git/hooks/pre-commit`:
+
+```bash
+#!/bin/sh
+npm run check-links:quick
+if [ $? -ne 0 ]; then
+ echo "❌ Broken links detected. Please fix before committing."
+ exit 1
+fi
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"Cannot read properties of null"**
+ - Ensure you're running the script from the project root directory
+ - Check that the `docs` directory exists
+
+2. **"No files found"**
+ - Verify the documentation directory path
+ - Ensure there are `.md` or `.mdx` files in the directory
+
+3. **"Permission denied"**
+ - Make sure the scripts have execute permissions:
+ ```bash
+ chmod +x tools/check-broken-links.js
+ chmod +x tools/quick-check-links.js
+ ```
+
+### Performance Tips
+
+- Use quick check for regular development
+- Run detailed check before releases
+- Consider excluding large directories if not needed
+- Use CI/CD integration for automated checking
+
+## Script Comparison
+
+| Feature | Quick Check | Detailed Check |
+|---------|-------------|----------------|
+| Speed | ⚡ Fast | 🐌 Thorough |
+| Line Numbers | ❌ No | ✅ Yes |
+| Link Text | ❌ No | ✅ Yes |
+| JSON Export | ❌ No | ✅ Yes |
+| Fix Suggestions | ❌ Basic | ✅ Detailed |
+| Excel Export | ✅ Yes | ✅ Yes |
+| Use Case | Development | Release/CI |
+
+## Contributing
+
+To improve the broken links checker:
+
+1. Fork the repository
+2. Make your changes to the scripts in the `tools/` directory
+3. Test with your documentation
+4. Submit a pull request
+
+## License
+
+This tool is part of your Docusaurus project and follows the same license.
\ No newline at end of file
diff --git a/tools/check-broken-links.js b/tools/check-broken-links.js
new file mode 100644
index 00000000..59aab3d5
--- /dev/null
+++ b/tools/check-broken-links.js
@@ -0,0 +1,369 @@
+#!/usr/bin/env node
+
+const fs = require('fs');
+const path = require('path');
+
+/**
+ * Broken Link Checker for Docusaurus
+ * Detects broken internal links in Docusaurus documentation and exports results to Excel
+ */
+
+class BrokenLinkChecker {
+ constructor(docsDir = './docs') {
+ this.docsDir = docsDir;
+ this.allFiles = new Set();
+ this.brokenLinks = [];
+ this.validExtensions = ['.md', '.mdx'];
+ }
+
+ /**
+ * Recursively get all files in directory
+ */
+ getFilesRecursively(dir, fileList = []) {
+ const files = fs.readdirSync(dir);
+
+ files.forEach(file => {
+ const filePath = path.join(dir, file);
+ const stat = fs.statSync(filePath);
+
+ if (stat.isDirectory()) {
+ this.getFilesRecursively(filePath, fileList);
+ } else if (this.validExtensions.includes(path.extname(file))) {
+ fileList.push(filePath);
+ }
+ });
+
+ return fileList;
+ }
+
+ /**
+ * Get all documentation files
+ */
+ getAllFiles() {
+ const files = this.getFilesRecursively(this.docsDir);
+
+ files.forEach(file => {
+ // Convert to relative path
+ const relativePath = path.relative(this.docsDir, file);
+ this.allFiles.add(relativePath);
+
+ // Add path without extension (Docusaurus supports this)
+ const withoutExt = relativePath.replace(/\.(md|mdx)$/, '');
+ this.allFiles.add(withoutExt);
+
+ // Add directory path for README.md files
+ if (path.basename(file) === 'README.md') {
+ const dirPath = path.dirname(relativePath);
+ if (dirPath !== '.') {
+ this.allFiles.add(dirPath);
+ }
+ }
+ });
+ }
+
+ /**
+ * Check if file exists
+ */
+ fileExists(linkPath, currentFile) {
+ const currentDir = path.dirname(currentFile);
+
+ // Handle relative paths
+ let targetPath;
+ if (linkPath.startsWith('./') || linkPath.startsWith('../')) {
+ targetPath = path.normalize(path.join(currentDir, linkPath));
+ } else if (linkPath.startsWith('/')) {
+ // Absolute path (relative to docs root)
+ targetPath = linkPath.substring(1);
+ } else {
+ // Relative path (not starting with ./)
+ targetPath = path.normalize(path.join(currentDir, linkPath));
+ }
+
+ // Normalize path separators
+ targetPath = targetPath.replace(/\\/g, '/');
+
+ // Check various possible paths
+ const possiblePaths = [
+ targetPath,
+ targetPath + '.md',
+ targetPath + '.mdx',
+ path.join(targetPath, 'README.md'),
+ path.join(targetPath, 'README.mdx')
+ ];
+
+ return possiblePaths.some(p => {
+ const normalizedPath = p.replace(/\\/g, '/');
+ return this.allFiles.has(normalizedPath) ||
+ fs.existsSync(path.join(this.docsDir, normalizedPath));
+ });
+ }
+
+ /**
+ * Extract links from file content
+ */
+ extractLinks(content) {
+ const links = [];
+
+ // Markdown link format: [text](link)
+ const markdownLinkRegex = /\[([^\]]*)\]\(([^)]+)\)/g;
+ let match;
+
+ while ((match = markdownLinkRegex.exec(content)) !== null) {
+ const linkText = match[1];
+ const linkUrl = match[2];
+
+ // Only process relative path links, exclude external links and anchors
+ if (!linkUrl.startsWith('http') &&
+ !linkUrl.startsWith('mailto:') &&
+ !linkUrl.startsWith('#') &&
+ !linkUrl.includes('://')) {
+
+ // Remove anchor part
+ const cleanUrl = linkUrl.split('#')[0];
+ if (cleanUrl) {
+ links.push({
+ text: linkText,
+ url: cleanUrl,
+ fullUrl: linkUrl,
+ line: this.getLineNumber(content, match.index)
+ });
+ }
+ }
+ }
+
+ // HTML link format:
+ const htmlLinkRegex = /
+ {/* Multi-Tables */}
+
+
+
+
+ Multi-Tables
+ Zero Impact
+
+
+
+
+
+ users
+
+ user_id
+
+
+ user_name
+
+
+ user_level
+
+
+ country
+
+
+ city
+
+
+ signup_time
+
+
+
+ orders
+
+ order_id
+
+
+ user_id
+
+
+ order_status
+
+
+ order_amount
+
+
+ payment_method
+
+
+ order_time
+
+
+ ...
+
+
+
+
+ {/* Processing */}
+
+
+
+
+
+
+
+
+
+ Real-time CDC
+
+
+
+
+
+
+ TapData
+ Operational Data Hub
+ Incremental Processing
+
+
+
+ {/* Materialized View */}
+
+
+
+
+
+
+
+
+
+ Incremental Materialized View
+
+
+
+ One Unified View
+
+ LIVE
+
+
+
+ ✓
+ Always up-to-date
+
+
+ ✓
+ No complex joins required
+
+
+ ✓
+ Fast, analysis-ready data
+
+
+ ✓
+ Power your BI and apps instantly
+
+
+
+
+ {/* Applications */}
+
+
+
+
+
+
+
+
+ Typical Applications
+
+
+
+ 📊
+ BI Reports
+
+
+ 🔌
+ Kafka
+
+
+ 🗄️
+ Databases
+
+
+ 🤖
+ AI MCP
+
+
+ 🔗
+ API
+
+