Skip to content

add load tsfile in data import #716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CritasWang merged 1 commit into from
May 15, 2025
Merged

add load tsfile in data import #716

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 131 additions & 17 deletions src/UserGuide/Master/Table/Tools-System/Data-Import-Tool.md
View file
Open in desktop

Large diffs are not rendered by default.

92 changes: 74 additions & 18 deletions src/UserGuide/Master/Tree/Tools-System/Data-Import-Tool.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Data Import

## 1. Overview
IoTDB supports two methods for data import:
* Data Import Tool: Use the import-data.sh (Unix/OS X) or import-data.bat (Windows) script in the tools directory to manually import CSV, SQL, or TsFile (open-source time-series file format) data into IoTDB.
* TsFile Auto-Loading Feature
IoTDB supports three methods for data import:
- Data Import Tool: Use the `import-data.sh/bat` script in the `tools` directory to manually import CSV, SQL, or TsFile (open-source time-series file format) data into IoTDB.
- `TsFile` Auto-Loading Feature
- Load `TsFile` SQL

<table style="text-align: left;">
<tbody>
Expand All @@ -21,12 +22,16 @@ IoTDB supports two methods for data import:
<td>Can be used for single or batch import of SQL files into IoTDB</td>
</tr>
<tr>
<td rowspan="2">TsFile</td>
<td rowspan="3">TsFile</td>
<td>Can be used for single or batch import of TsFile files into IoTDB</td>
</tr>
<tr>
<td>TsFile Auto-Loading Feature</td>
<td>Can automatically monitor a specified directory for newly generated TsFiles and load them into IoTDB.</td>
<td>TsFile Auto-Loading Feature</td>
<td>Can automatically monitor a specified directory for newly generated TsFiles and load them into IoTDB</td>
</tr>
<tr>
<td>Load SQL</td>
<td>Can be used for single or batch import of TsFile files into IoTDB</td>
</tr>
</tbody>
</table>
Expand All @@ -46,9 +51,9 @@ IoTDB supports two methods for data import:
| `-tz` | `--timezone` | Timezone (e.g., `+08:00`, `-01:00`). | No | System default |
| `-help` | `--help` | Display help (general or format-specific: `-help csv`). | No | - |

### 2.2 CSV Format
### 2.2 CSV Format

#### 2.2.1 Command
#### 2.2.1 Command
```Shell
# Unix/OS X
> tools/import-data.sh -ft<format> [-h <host>] [-p <port>] [-u <username>] [-pw <password>]
Expand All @@ -63,7 +68,7 @@ IoTDB supports two methods for data import:
[-tn <thread_num>]
```

#### 2.2.2 CSV-Specific Parameters
#### 2.2.2 CSV-Specific Parameters

| Short | Full Parameter | Description | Required | Default |
| ---------------- | ------------------------------- |----------------------------------------------------------| ---------- |-----------------|
Expand All @@ -74,7 +79,7 @@ IoTDB supports two methods for data import:
| `-ti` | `--type_infer` | Type mapping (e.g., `BOOLEAN=text,INT=long`). | No | - |
| `-tp` | `--timestamp_precision` | Timestamp precision: `ms`, `us`, `ns`. | No | `ms` |

#### 2.2.3 Examples
#### 2.2.3 Examples

```Shell
# Valid Example
Expand Down Expand Up @@ -134,9 +139,9 @@ Time,Device,str(TEXT),var(INT32)
```


### 2.3 SQL Format
### 2.3 SQL Format

#### 2.3.1 Command
#### 2.3.1 Command

```Shell
# Unix/OS X
Expand All @@ -150,15 +155,15 @@ Time,Device,str(TEXT),var(INT32)
[-batch <batch_size>] [-tn <thread_num>]
```

#### 2.3.2 SQL-Specific Parameters
#### 2.3.2 SQL-Specific Parameters

| Short | Full Parameter | Description | Required | Default |
| -------------- | ------------------------------- | -------------------------------------------------------------------- | ---------- | ------------------ |
| `-fd` | `--fail_dir` | Directory to save failed files. | No |YOUR_CSV_FILE_PATH|
| `-lpf` | `--lines_per_failed_file` | Max lines per failed file. | No | `100000` <br> Range: 0 to Integer.Max(2147483647). |
| `-batch` | `--batch_size` | Rows processed per API call. | No | `100000` <br> Range: 0 to Integer.Max(2147483647). |

#### 2.3.3 Examples
#### 2.3.3 Examples

```Shell
# Valid Example
Expand All @@ -174,9 +179,9 @@ error: Source file or directory /path/sql does not exist
> tools/import-data.sh -ft sql -s /path/sql -tn 0
error: Invalid thread number '0'. Please set a positive integer.
```
### 2.4 TsFile Format
### 2.4 TsFile Format

#### 2.4.1 Command
#### 2.4.1 Command

```Shell
# Unix/OS X
Expand All @@ -189,7 +194,7 @@ error: Invalid thread number '0'. Please set a positive integer.
-s <source> -os <on_success> [-sd <success_dir>] -of <on_fail> [-fd <fail_dir>]
[-tn <thread_num> ] [-tz <timezone>] [-tp <timestamp precision (ms/us/ns)>]
```
#### 2.4.2 TsFile-Specific Parameters
#### 2.4.2 TsFile-Specific Parameters

| Short | Full Parameter | Description | Required | Default |
| ----------- | ----------------------------- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- | --------------------------- |
Expand All @@ -199,7 +204,7 @@ error: Invalid thread number '0'. Please set a positive integer.
| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` |
| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`. <br> For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned. <br> ​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` |

#### 2.4.3 Examples
#### 2.4.3 Examples

```Shell
# Valid Example
Expand Down Expand Up @@ -242,3 +247,54 @@ Add the following parameters to `iotdb-system.properties` (template: `iotdb-syst
2. ​​**Restricted Directories**​: Do NOT set Pipe receiver directories, data directories, or other system paths as monitored directories.
3. ​​**Directory Conflicts**​: Ensure `load_active_listening_fail_dir` does not overlap with `load_active_listening_dirs` or its subdirectories.
4. ​​**Permissions**​: The monitored directory must have write permissions. Files are deleted after successful loading; insufficient permissions may cause duplicate loading.

## 4. Load SQL

IoTDB supports importing one or multiple TsFile files containing time series into another running IoTDB instance directly via SQL execution through the CLI.

### 4.1 Command

```SQL
load '<path/dir>' with (
'attribute-key1'='attribute-value1',
'attribute-key2'='attribute-value2',
)
```

* `<path/dir>` : The path to a TsFile or a folder containing multiple TsFiles.
* `<attributes>`: Optional parameters, as described below.

| Key | Key Description | Value Type | Value Range | Value is Required | Default Value |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|--------------------------------|-------------------|----------------------------|
| `database-level` | When the database corresponding to the TsFile does not exist, the database hierarchy level can be specified via the ` database-level` parameter. The default is the level set in `iotdb-common.properties`. For example, setting level=1 means the prefix path of level 1 in all time series in the TsFile will be used as the database. | Integer | `[1: Integer.MAX_VALUE]` | No | 1 |
| `on-success` | Action for successfully loaded TsFiles: `delete` (delete the TsFile after successful import) or `none` (retain the TsFile in the source folder). | String | `delete / none` | No | delete |
| `model` | Specifies whether the TsFile uses the `table` model or `tree` model. | String | `tree / table` | No | Aligns with `-sql_dialect` |
| `database-name` | Table model only: Target database for import. Automatically created if it does not exist. The database-name must not include the `root.` prefix (an error will occur if included). | String | `-` | No | null |
| `convert-on-type-mismatch` | Whether to perform type conversion during loading if data types in the TsFile mismatch the target schema. | Boolean | `true / false` | No | true |
| `verify` | Whether to validate the schema before loading the TsFile. | Boolean | `true / false` | No | true |
| `tablet-conversion-threshold` | Size threshold (in bytes) for converting TsFiles into tablet format during loading. Default: `-1` (no conversion for any TsFile). | Integer | `[-1,0 :`​`Integer.MAX_VALUE]` | No | -1 |
| `async` | Whether to enable asynchronous loading. If enabled, TsFiles are moved to an active-load directory and loaded into the `database-name` asynchronously. | Boolean | `true / false` | No | false |

### 4.2 Example

```SQL
-- Before import
IoTDB> show databases
+-------------+-----------------------+---------------------+-------------------+---------------------+
| Database|SchemaReplicationFactor|DataReplicationFactor|TimePartitionOrigin|TimePartitionInterval|
+-------------+-----------------------+---------------------+-------------------+---------------------+
|root.__system| 1| 1| 0| 604800000|
+-------------+-----------------------+---------------------+-------------------+---------------------+

-- Import tsfile by excuting load sql
IoTDB> load '/home/dump1.tsfile' with ( 'on-success'='none')
Msg: The statement is executed successfully.

-- Verify whether the import was successful
IoTDB> select * from root.testdb.**
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
| Time|root.testdb.device.model.temperature|root.testdb.device.model.humidity|root.testdb.device.model.status|
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
|2025-04-17T10:35:47.218+08:00| 22.3| 19.4| true|
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
```
Loading