Skip to main content
Version: Next

TDengine

TDengine sink connector

Support Those Engines​

Spark
Flink
SeaTunnel Zeta

Description​

Write data to TDengine.

Create the target database and super table before running the SeaTunnel job. The sink can write one input table to one super table, or use placeholders such as ${table_name} in stable for multi-table writes.

The input row must follow TDengine's super table write shape: the first field is the target sub table name, the middle fields are normal columns, and the last fields are TAGS values. The connector reads the number of TAGS fields from the target super table metadata.

For example, if the target super table has two TAGS fields, the last two input fields are treated as TAGS values, and the first input field is treated as the sub table name.

Key features​

Options​

NameTypeRequiredDefaultDescription
urlStringYes-TDengine REST JDBC URL. For example jdbc:TAOS-RS://localhost:6041/.
usernameStringYes-Username used to connect to TDengine.
passwordStringYes-Password used to connect to TDengine.
databaseStringYes-TDengine database name.
stableStringYes-TDengine super table name. For multi-table writes, this value can contain placeholders such as ${table_name}.
timezoneStringNoUTCTDengine server timezone used for timestamp conversion.
write_columnsListNo-Normal TDengine column names to insert. When unset, the target super table column order is used. Do not include TAGS columns or the sub-table-name field.
common-optionsNo-Sink plugin common parameters. See Sink Common Options.

url [String]​

The TDengine REST JDBC URL.

For example:

jdbc:TAOS-RS://localhost:6041/

username [String]​

The username used to connect to TDengine.

password [String]​

The password used to connect to TDengine.

database [String]​

The TDengine database name. The database must already exist on the server.

stable [String]​

The TDengine super table name. The value is used verbatim by the sink writer; the TDengine connector itself does not perform placeholder substitution, so ${table_name} and similar tokens are not rewritten at runtime. For multi-table writes, SeaTunnel's upstream framework (TablePlaceholderProcessor) may substitute stable once during job setup based on the upstream CatalogTable identifier, but this depends on the upstream framework wiring and is not a TDengine-specific feature.

timezone [String]​

The TDengine server timezone used for timestamp conversion. The default value is UTC. Set this to match your TDengine server if it is not running in UTC.

write_columns [List]​

The normal TDengine column names to insert. If it is not set, TDengine uses the column order of the target super table. Do not include the first input field that contains the sub table name, and do not include TAGS columns; the connector adds TAGS values from the end of the input row automatically.

common options​

Sink plugin common parameters, please refer to Sink Common Options for details. For multi-table writes, multi_table_sink_replica can be used with the common sink options.

Input Row Shape​

The connector expects every input row to follow the super-table write shape:

  1. The first field is the target sub table name (string). The sink creates the sub table on demand if it does not exist.
  2. The next fields are the normal columns declared in write_columns (or in the target super table column order when write_columns is unset).
  3. The last fields are TAGS values. The number of TAGS fields is read from the target super table metadata.

If the target super table has two TAGS fields, the last two input fields are TAGS values and the first input field is the sub table name.

Examples​

Write to one super table​

env {
  parallelism = 2
  job.mode = "BATCH"
}

sink {
  TDengine {
    url = "jdbc:TAOS-RS://localhost:6041/"
    username = "root"
    password = "taosdata"
    database = "power2"
    stable = "meters2"
    timezone = "UTC"
    write_columns = ["ts", "voltage", "current", "power"]
  }
}

Write multiple input tables to matching super tables​

source {
  FakeSource {
    plugin_output = "fake"
    tables_configs = [
      {
        schema = {
          table = "meters3"
          fields {
            device_id = "string"
            event_time = "timestamp"
            metric1 = "float"
            metric2 = "int"
            metric3 = "float"
            status_flag = "boolean"
            notes = "string"
            location_tag = "string"
            group_tag = "int"
          }
        }
        rows = [
          {
            kind = INSERT
            fields = ["d2001", "2023-04-22T14:38:05", 10.3, 219, 0.31, true, "nc", "California.SanFrancisco", 2]
          }
        ]
      },
      {
        schema = {
          table = "meters4"
          fields {
            device_id = "string"
            event_time = "timestamp"
            metric1 = "float"
            metric2 = "int"
            metric3 = "float"
            status_flag = "boolean"
            notes = "string"
            location_tag = "string"
            group_tag = "int"
          }
        }
        rows = [
          {
            kind = INSERT
            fields = ["d1005", "2023-04-22T14:38:05", 110.3, 219, 0.31, true, "nc", "California.SanFrancisco", 2]
          }
        ]
      }
    ]
  }
}

sink {
  TDengine {
    url = "jdbc:TAOS-RS://localhost:6041/"
    username = "root"
    password = "taosdata"
    database = "power2"
    stable = "${table_name}"
    timezone = "UTC"
  }
}

Here ${table_name} is treated as a literal string by the TDengine sink writer (the connector does not substitute it per row), so this example only works if the upstream framework substitutes stable once at job setup with the CatalogTable identifier from the upstream source. The target super tables must already exist with matching TAGS columns.

Changelog​

Change Log
ChangeCommitVersion
[Feature][connector-tdengine] Support subtable and fieldNames in tdengine source (#9593)https://github.com/apache/seatunnel/commit/b136a0dc432.3.12
[improve] tdengine options (#9399)https://github.com/apache/seatunnel/commit/ff122fe4052.3.12
[Feature][Connector-V2] Support multi-table sink feature for TDengine (#9215)https://github.com/apache/seatunnel/commit/98b593f0952.3.11
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Fix][Connector-V2] Fix NullPointerException when column or tag contains null value in TDengine sink (#9158)https://github.com/apache/seatunnel/commit/a047cab5462.3.11
[Fix][Connector][TDEngine] TDEngine support NCHAR type (#8411)https://github.com/apache/seatunnel/commit/88c92ae1b12.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Improve][Connector-V2] Close all ResultSet after used (#7389)https://github.com/apache/seatunnel/commit/853e9732122.3.8
[Fix][Connector-tdengine] Fix sql exception and concurrentmodifyexception when connect to taos and read datahttps://github.com/apache/seatunnel/commit/a18fca80062.3.7
[Bugfix][TDengine] Fix the issue of losing the driver due to multiple calls to the submit job REST API #6581 (#6596)https://github.com/apache/seatunnel/commit/470bb974342.3.5
[improve][connector-tdengine] support read bool column from tdengine (#6025)https://github.com/apache/seatunnel/commit/af39235ee32.3.4
[Bugfix][TDengine] Fix the degree of multiple parallelism affects driver loading (#6020)https://github.com/apache/seatunnel/commit/b6ebbd47b22.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)https://github.com/apache/seatunnel/commit/51c0d709ba2.3.4
[Hotfix][Connector] Fixed TDengine connector using jdbc driver to cause loading error (#4598)https://github.com/apache/seatunnel/commit/78f7989b812.3.2
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector-V2] add tdengine source (#2832)https://github.com/apache/seatunnel/commit/acf4d5b1b42.3.1