TapData Connectors

TapData Connectors is an open-source project for developing and building custom data connectors, designed for the TapData data synchronization platform. Through this project, developers can easily build custom data connectors and register them with the TapData platform, enabling rapid integration of various heterogeneous data sources.

Why Choose TapData Connectors?

• Rich Connector Ecosystem: Supports mainstream databases, cloud services, SaaS applications, and more
• Rapid Development: Based on the PDK framework, new connectors can be developed in just a few hours
• Enterprise Features: Supports incremental synchronization, resume from breakpoint, and more
• Cross-Platform Support: Compatible with Windows, Linux, macOS, and other mainstream operating systems
• High Performance: Optimized data transmission engine supporting real-time synchronization of large data volumes

🚀 Quick Start

# 1️⃣ Clone the project
git clone https://github.com/tapdata/tapdata-connectors.git && cd tapdata-connectors

# 2️⃣ Build MySQL connector (example)
mvn clean install -pl connectors/mysql-connector -am -DskipTests

# 3️⃣ Register with TapData platform
java -jar pdk-deploy.jar register \
  -a YOUR_ACCESS_CODE \
  -t http://localhost:3030 \
  connectors/mysql-connector/target/mysql-connector-*.jar

✅ Done! Now you can use this connector in the TapData platform.

Project Structure

tapdata-connectors/
├── connectors                # Various Java connectors
├── connectors-common         # Common dependencies and Debezium aggregation modules
├── connectors-javascript     # JavaScript connectors and core
├── connectors-unpackage      # Unpackaged special connectors
├── connectors-tdd            # Development/test-driven demonstrations
├── file-storages             # File storage connectors
├── tapdata-cli               # CLI tools

Development Environment Setup

System Requirements

• JDK: 17 or higher
• Maven: 3.6+
• Operating System: Windows / Linux / macOS

Environment Configuration

Linux

# Ubuntu/Debian
sudo apt-get install openjdk-17-jdk maven

# CentOS/RHEL
sudo yum install java-17-openjdk-devel maven

macOS

# Auto-detect and set JDK 17
export JAVA_HOME=$(/usr/libexec/java_home -v17 2>/dev/null || echo "/usr/lib/jvm/java-17-openjdk")
export PATH=$JAVA_HOME/bin:$PATH
java -version

Windows

1. Download and install JDK 17
2. Download and extract Maven
3. Configure environment variables JAVA_HOME and MAVEN_HOME

Build Connectors

mvn clean install -DskipTests \
  -pl 'module-name'
  -am

Optional Parameters:

• -DskipTests: Skip tests, recommended during compilation to improve build speed
• -pl: Build only selected modules; to exclude certain modules, use exclusion syntax in -pl, e.g., !module-name1,!module-name2
• -am: Automatically build dependencies of selected modules

After compilation, you can find the compiled connector JAR files in the corresponding module's target/*.jar. Then follow the instructions below to register them with the TapData platform.

Register Connector to TapData Platform

java -jar pdk-deploy.jar register -a ${access_code} -t ${tm_url} \
  [-ak ${accessKey} [-sk ${secretKey}]] [-r ${oem_type}] [-f ${filter_type}] [-l] [-h] [-X] \
  /path/to/your-connector.jar

Parameter Description:

• -a (--auth): The access_code in TapData. Log in to the TapData management platform, click your username in the upper right corner, select Personal Settings to view it
• -t (--tm): TapData management platform login address, e.g., http://localhost:3030
• -f (--filter): Register only connectors of specified authentication types, multiple values separated by commas
• -l (--latest): Replace with the latest version
• -h (--help): View command help information

The pdk-deploy.jar tool can be found in your TapData deployment environment's tapdata/apps/lib directory.

Execution example:

java -jar pdk-deploy.jar register \
  -a 3324***********8d4562f \
  -t http://127.0.0.1:3030 \
  connectors/starrocks-connector/target/starrocks-connector-1.0-SNAPSHOT.jar

When the connector is successfully registered, it will prompt "Completed". Then you can log in to the TapData management page and use this connector when creating connections.

FAQ

Q: Getting protoc compilation errors when building on macOS?

A: For Apple Silicon (macOS ARM), it's recommended to exclude debezium-connector-postgres and debezium-connector-highgo to avoid local binary compatibility issues (see module list in connectors-common/debezium-bucket/pom.xml:71-81).

Build command example:

mvn clean install -DskipTests \
  -pl '!connectors-common/debezium-bucket/debezium-connector-postgres,!connectors-common/debezium-bucket/debezium-connector-highgo'
  -am

How to Contribute

We warmly welcome more contributors to help improve and expand connectors! Every contribution matters, no matter how big or small!

Contribution Methods

Contribution Type	Description	Difficulty
Report Bugs	Found a bug? Feel free to submit an Issue	⭐
Improve Documentation	Fix typos, add examples, optimize descriptions	⭐
New Features	Develop new connectors or enhance existing functionality	⭐⭐⭐
Code Optimization	Performance optimization, code refactoring	⭐⭐
Test Cases	Add test cases to improve code quality	⭐⭐

PR Template Suggestions

• Title: Clear and concise, e.g., Add MongoDB connector support
• Description: Explain the background, scope of changes, and test results
• Checklist:
  • Code passes local testing
  • Documentation has been updated
  • Follows code standards

Contact Us

Encountered issues? Want to connect? We provide multiple support channels!

Instant Communication

Technical Community

Reference Documentation

• TapData Community Edition Open Source Repository
• TapData Online Documentation

🎉 Acknowledgments

Thank you to all developers, users, and supporters who have contributed to TapData Connectors. It's with your support that this project continues to grow and thrive.

If this project has been helpful to you, please give us a Star!

Contributors