Skip to main content
Version: Next

TDengine

TDengine sink connector

Description​

Write data to TDengine.

Create the target database and super table before running the SeaTunnel job. The sink can write one input table to one super table, or use placeholders such as ${table_name} in stable for multi-table writes.

The input row must follow TDengine's super table write shape: the first field is the target sub table name, the middle fields are normal columns, and the last fields are TAGS values. The connector reads the number of TAGS fields from the target super table metadata.

For example, if the target super table has two TAGS fields, the last two input fields are treated as TAGS values, and the first input field is treated as the sub table name.

Key features​

Options​

nametyperequireddefault value
urlstringyes-
usernamestringyes-
passwordstringyes-
databasestringyes-
stablestringyes-
timezonestringnoUTC
write_columnslistno-
common-optionsno-

url [string]​

The TDengine REST JDBC URL.

e.g.

jdbc:TAOS-RS://localhost:6041/

username [string]​

The username used to connect to TDengine.

password [string]​

The password used to connect to TDengine.

database [string]​

The TDengine database name.

stable [string]​

The TDengine super table name. For multi-table writes, this value can contain placeholders, for example ${table_name}.

timezone [string]​

The TDengine server timezone used for timestamp conversion. The default value is UTC.

write_columns [list]​

The normal TDengine column names to insert. If it is not set, TDengine uses the column order of the target super table. Do not include the first input field that contains the sub table name, and do not include TAGS columns; the connector adds TAGS values from the end of the input row automatically.

common options​

Sink plugin common parameters, please refer to Sink Common Options for details. For multi-table writes, multi_table_sink_replica can be used with the common sink options.

Examples​

Write to one super table​

env {
  parallelism = 2
  job.mode = "BATCH"
}

sink {
  TDengine {
    url = "jdbc:TAOS-RS://localhost:6041/"
    username = "root"
    password = "taosdata"
    database = "power2"
    stable = "meters2"
    timezone = "UTC"
    write_columns = ["ts", "voltage", "current", "power"]
  }
}

Write multiple input tables to matching super tables​

source {
  FakeSource {
    plugin_output = "fake"
    tables_configs = [
      {
        schema = {
          table = "meters3"
          fields {
            device_id = "string"
            event_time = "timestamp"
            metric1 = "float"
            metric2 = "int"
            metric3 = "float"
            status_flag = "boolean"
            notes = "string"
            location_tag = "string"
            group_tag = "int"
          }
        }
        rows = [
          {
            kind = INSERT
            fields = ["d2001", "2023-04-22T14:38:05", 10.3, 219, 0.31, true, "nc", "California.SanFrancisco", 2]
          }
        ]
      },
      {
        schema = {
          table = "meters4"
          fields {
            device_id = "string"
            event_time = "timestamp"
            metric1 = "float"
            metric2 = "int"
            metric3 = "float"
            status_flag = "boolean"
            notes = "string"
            location_tag = "string"
            group_tag = "int"
          }
        }
        rows = [
          {
            kind = INSERT
            fields = ["d1005", "2023-04-22T14:38:05", 110.3, 219, 0.31, true, "nc", "California.SanFrancisco", 2]
          }
        ]
      }
    ]
  }
}

sink {
  TDengine {
    url = "jdbc:TAOS-RS://localhost:6041/"
    username = "root"
    password = "taosdata"
    database = "power2"
    stable = "${table_name}"
    timezone = "UTC"
  }
}

Changelog​

Change Log
ChangeCommitVersion
[Feature][connector-tdengine] Support subtable and fieldNames in tdengine source (#9593)https://github.com/apache/seatunnel/commit/b136a0dc432.3.12
[improve] tdengine options (#9399)https://github.com/apache/seatunnel/commit/ff122fe4052.3.12
[Feature][Connector-V2] Support multi-table sink feature for TDengine (#9215)https://github.com/apache/seatunnel/commit/98b593f0952.3.11
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Fix][Connector-V2] Fix NullPointerException when column or tag contains null value in TDengine sink (#9158)https://github.com/apache/seatunnel/commit/a047cab5462.3.11
[Fix][Connector][TDEngine] TDEngine support NCHAR type (#8411)https://github.com/apache/seatunnel/commit/88c92ae1b12.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Improve][Connector-V2] Close all ResultSet after used (#7389)https://github.com/apache/seatunnel/commit/853e9732122.3.8
[Fix][Connector-tdengine] Fix sql exception and concurrentmodifyexception when connect to taos and read datahttps://github.com/apache/seatunnel/commit/a18fca80062.3.7
[Bugfix][TDengine] Fix the issue of losing the driver due to multiple calls to the submit job REST API #6581 (#6596)https://github.com/apache/seatunnel/commit/470bb974342.3.5
[improve][connector-tdengine] support read bool column from tdengine (#6025)https://github.com/apache/seatunnel/commit/af39235ee32.3.4
[Bugfix][TDengine] Fix the degree of multiple parallelism affects driver loading (#6020)https://github.com/apache/seatunnel/commit/b6ebbd47b22.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)https://github.com/apache/seatunnel/commit/51c0d709ba2.3.4
[Hotfix][Connector] Fixed TDengine connector using jdbc driver to cause loading error (#4598)https://github.com/apache/seatunnel/commit/78f7989b812.3.2
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector-V2] add tdengine source (#2832)https://github.com/apache/seatunnel/commit/acf4d5b1b42.3.1