Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/actions/spelling/allow.txt
Original file line number Diff line number Diff line change
Expand Up @@ -776,4 +776,7 @@ czf
mongodump
mongorestore
OBJEXIST
tdmysql
tdsql
tdsqld

135 changes: 88 additions & 47 deletions docs/connectors/cloud-databases/tencentdb-for-mysql.md
Original file line number Diff line number Diff line change
@@ -1,78 +1,119 @@
# TencentDB TD-SQL
---
pdkId: tencent-db-mysql
---

# TDSQL (MySQL Edition)

TDSQL (MySQL Edition), referred to as TDSQL MySQL in this document, is a distributed database service on Tencent Cloud. It supports automatic sharding and a shared-nothing architecture. TapData can use TDSQL MySQL as either a source or target to build data pipelines for cloud migration and real-time replication.

Please follow the instructions below to ensure successful addition and use of the distributed database TD-SQL version database in TapData.
```mdx-code-block
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
```

## Supported Versions

TencentDB TD-SQL 5.7.x, 8.0.x
MySQL kernel versions **5.7** and **8.0**.

## As a Data Source

* Enable the Binlog feature on the source database.
## SQL Operations for Sync

- Create an Account
- **DML**: INSERT, UPDATE, DELETE

For MySQL 8 and later, password encryption is different. Make sure to use the corresponding method for your version to set the password; otherwise, incremental synchronization may fail. Use the following commands to confirm whether supplemental logging is enabled.
:::tip

**For 5.x Versions**
When TDSQL MySQL is used as a target, you can also choose a write policy in the node advanced settings. For insert conflicts, you can update or discard; for update failures, you can insert instead or only log the error.

```sql
CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
```
:::

**For 8.x Versions**
- **DDL**: ADD COLUMN, CHANGE COLUMN (auto-increment not supported), DROP COLUMN, RENAME COLUMN

```sql
-- Create the user
CREATE USER 'username'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';
-- Change the password
ALTER USER 'username'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';
```
## Prerequisites

### Granting Permissions
To connect to TDSQL MySQL (as either a source or target), follow these steps. TDSQL MySQL enables binlog by default, so no additional setup is required for incremental sync when it is used as a source.

Grant `SELECT` permissions for a specific database:
1. Go to the [TDSQL instance list](https://console.intl.cloud.tencent.com/tdsqld/instance-tdmysql), select a region, and click the target instance ID.

```sql
GRANT SELECT, SHOW VIEW, CREATE ROUTINE, LOCK TABLES ON <DATABASE_NAME>.<TABLE_NAME> TO 'tapdata' IDENTIFIED BY 'password';
```
2. Create an account and grant privileges.

1. On the instance management page, open **Account Management** and click **Create Account**.
2. In the dialog, enter the account name, host, password, etc., then click **Confirm, Next**.
The host value is the network egress address. Use `%` to allow access from any IP.
3. Click **Modify Privileges** next to the account, select the required privileges, and click **Confirm**.
![Grant privileges](../../images/grant_tdsql_permission.png)

Grant global privileges:
<Tabs className="unique-tabs">
<TabItem value="As Source" default>

```sql
GRANT RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'tapdata' IDENTIFIED BY 'password';
```
* **Full sync only**: Grant `SELECT` on the database to be synced
* **Full + incremental sync**: Grant `SELECT` on the database to be synced, plus replication privileges (`REPLICATION SLAVE`, `REPLICATION CLIENT`)

</TabItem>

<TabItem value="As Target">

### Constraint Explanation
Grant read/write privileges on the target database, including `ALTER`, `CREATE`, `CREATE ROUTINE`, `CREATE TEMPORARY TABLES`, `DELETE`, `DROP`, `INSERT`, `SELECT`, and `UPDATE`.

</TabItem>

</Tabs>

When synchronizing from MySQL to other heterogeneous databases, if the source MySQL database has table-level cascade settings, data updates and deletes triggered by this cascade will not be propagated to the target. If you need to build cascading processing capabilities on the target side, you can use triggers or other methods to achieve this type of data synchronization.

## As a Target
3. On the **Instance Details** page, click **Enable** in the **Public Network Address** section and follow the prompts to obtain the public connection address.

Grant full privileges for a specific database:
![Get public connection address](../../images/obtain_tdsql_external_address.png)

```sql
GRANT ALL PRIVILEGES ON <DATABASE_NAME>.<TABLE_NAME> TO 'tapdata' IDENTIFIED BY 'password';
```
:::tip
If the TapData host and the TDSQL MySQL instance are in the same private network, you can skip this step.
:::

Grant global privileges:
## Connect to TDSQL MySQL

```sql
GRANT PROCESS ON *.* TO 'tapdata' IDENTIFIED BY 'password';
```
1. Log in to TapData.

### Common Errors
2. In the left navigation pane, click **Connection**.

"Unknown error 1044"
3. Click **Create**.

If permissions are granted correctly but you're still unable to pass the test connection through TapData, you can use the following steps to check and fix the issue:
4. In the dialog, search for and select **TDSQL(MySQL Edition)**.

5. On the configuration page, fill in the connection settings as follows.
![TDSQL MySQL connection settings](../../images/tdsql_connection_settings.png)

* Connection settings
* **Name**: Enter a unique name with business significance.
* **Type**: Specify whether TDSQL MySQL is used as a source or target.
* **Host**: The database address. Use the public address obtained in the prerequisites. If TapData and TDSQL MySQL are in the same private network, use the private address.
* **Port**: The database service port.
* **Database**: The database name. One connection maps to one database. Create multiple connections if you need to access multiple databases.
* **Username**: A database account with the required privileges.
* **Password**: The password for the database account.
* **Table Name Case Sensitive**: An initialization parameter for newly created distributed TDSQL MySQL instances (default: Insensitive). Use the value that matches your instance configuration.
* **Connection Parameter String**: Additional parameters. Default: `useUnicode=yes&characterEncoding=UTF-8`. Adjust based on your instance configuration, especially the character set.
* **Timezone**: Default is UTC+0. If you set a different timezone, it affects fields without timezone information (for example, `datetime`). Fields with timezone information (for example, `timestamp`) and `date`/`time` are not affected.
* Advanced settings
- **CDC Log Caching**: [Extract the incremental logs](../../operational-data-hub/advanced/share-mining.md) from the source database. This allows multiple tasks to share the incremental log extraction process from the same source, reducing the load on the source database. When enabled, you also need to select a storage location for the incremental log information.
- **Include Tables**: By default, all tables are included. You can choose to customize and specify the tables to include, separated by commas.
- **Exclude Tables**: When enabled, you can specify tables to exclude, separated by commas.
- **Agent Settings**: The default is automatic assignment by the platform. You can also manually specify an Agent.
- **Model Load Time**: If there are less than 10,000 models in the data source, their schema will be updated every hour. But if the number of models exceeds 10,000, the refresh will take place daily at the time you have specified.
- **Enable Heartbeat Table**: When the connection type is Source or Source and Target, enable this feature. TapData creates and periodically updates `_tapdata_heartbeat_table` in the source (requires write privileges) to monitor connection and task health. The heartbeat task is automatically enabled only after a task referencing the source starts. You can view its status on the edit page of the data source.

6. Click **Test**. After the test passes, click **Save**.

:::tip

If the connection test fails, follow the on-screen guidance to troubleshoot.

:::


## Node Advanced Features

When you use TDSQL MySQL as a source or target in a replication/transformation task, TapData provides additional advanced options to handle complex scenarios and improve performance.

![Advanced node capabilities](../../images/tdsql_node_advanced_settings.png)

```sql
SELECT host, user, Grant_priv, Super_priv FROM mysql.user WHERE user='username';
-- Check if the value of the Grant_priv field is 'Y'
-- If not, execute the following command
UPDATE mysql.user SET Grant_priv='Y' WHERE user='username'; FLUSH PRIVILEGES;
```

* **As a source node**: Supports **Full Breakpoint Resume**. For large-scale migrations (over 100 million records), you can enable this option to shard and migrate data, reducing the risk of failures caused by interruptions and improving reliability.
* **As a target node**: Choose the table creation type when the target table does not exist (single table or partitioned table). For details, see the [official TDSQL MySQL documentation](https://intl.cloud.tencent.com/document/product/1042/38506).
18 changes: 9 additions & 9 deletions docs/connectors/supported-data-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -544,6 +544,15 @@ The beta version of the data sources is in public preview and has passed the bas
<td>➖</td>
<td>N/A</td>
</tr>
<tr>
<td>TDSQL (MySQL Edition)</td>
<td>✅</td>
<td>✅</td>
<td>✅</td>
<td>✅</td>
<td>✅</td>
<td>5.7, 8.0</td>
</tr>
<tr>
<td>Zoho-CRM</td>
<td>✅</td>
Expand Down Expand Up @@ -859,15 +868,6 @@ The Alpha version of the data sources is in public preview and has passed the ba
<td>➖</td>
<td>3.2, 3.4, 3.6, 4.0, 4.2</td>
</tr>
<tr>
<td>TencentDB MySQL</td>
<td>✅</td>
<td>✅</td>
<td>➖</td>
<td>✅</td>
<td>➖</td>
<td>5.5, 5.6, 5.7, 8.x</td>
</tr>
<tr>
<td>TencentDB PostgreSQL</td>
<td>✅</td>
Expand Down
Binary file added docs/images/grant_tdsql_permission.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/obtain_tdsql_external_address.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/tdsql_connection_settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/tdsql_node_advanced_settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,9 @@ const sidebars = {
'connectors/cloud-databases/mongodb-atlas',
'connectors/cloud-databases/polardb-mysql',
'connectors/cloud-databases/polardb-postgresql',
'connectors/cloud-databases/tencentdb-for-mysql',
'connectors/cloud-databases/tencentdb-for-mariadb',
'connectors/cloud-databases/tencentdb-for-mongodb',
'connectors/cloud-databases/tencentdb-for-mysql',
'connectors/cloud-databases/tencentdb-for-pg',
'connectors/cloud-databases/tencentdb-for-sql-server',
]
Expand Down
Loading