Skip to main content
Version: Next

TDengine

TDengine source connector

Support Those Engines​

Spark
Flink
SeaTunnel Zeta

Description​

Read data from TDengine super tables.

The source reads data in batch mode by querying a time range from one super table. You can read all sub tables under the super table, limit the read to specific sub tables, and select only part of the columns.

Each source split reads one TDengine sub table. The output schema always adds subtable_name as the first field so that downstream sinks can keep the original TDengine sub table name. This is also the field consumed by the TDengine sink when writing rows back to TDengine.

Key features​

Options​

NameTypeRequiredDefaultDescription
urlStringYes-TDengine REST JDBC URL. For example jdbc:TAOS-RS://localhost:6041/.
usernameStringYes-Username used to connect to TDengine.
passwordStringYes-Password used to connect to TDengine.
databaseStringYes-TDengine database name.
stableStringYes-TDengine super table name.
sub_tablesListNo-Sub table names to read. When unset, all sub tables under stable are read.
lower_boundStringYes-Inclusive lower bound of the query time range. Use a TDengine-compatible timestamp such as 2018-10-03 14:38:05.000.
upper_boundStringYes-Exclusive upper bound of the query time range. Use a TDengine-compatible timestamp such as 2018-10-03 14:38:16.801.
read_columnsListNo-Columns to read. When unset, all columns are read. Put TAGS columns at the end of the list; do not include subtable_name.
common-optionsNo-Source plugin common parameters. See Source Common Options.

url [String]​

The TDengine REST JDBC URL.

For example:

jdbc:TAOS-RS://localhost:6041/

username [String]​

The username used to connect to TDengine.

password [String]​

The password used to connect to TDengine.

database [String]​

The TDengine database name. The database must already exist on the server.

stable [String]​

The TDengine super table name. The connector splits the read into one source split per sub table under this super table.

sub_tables [List]​

A list of sub table names. If it is not configured, all sub tables under the configured super table are read. If it is configured, only the listed sub tables are read. Sub table names must match the server exactly.

lower_bound [String]​

The inclusive lower bound of the query time range. The connector adds timestamp_column >= lower_bound to each sub table query. Use a TDengine-compatible timestamp string, for example 2018-10-03 14:38:05.000.

upper_bound [String]​

The exclusive upper bound of the query time range. The connector adds timestamp_column < upper_bound to each sub table query. Use a TDengine-compatible timestamp string, for example 2018-10-03 14:38:16.801.

read_columns [List]​

A list of column names to read. If it is not configured, all columns are read. When reading from a super table, put TAGS columns at the end of the list. Do not include subtable_name; the connector adds it automatically as the first output field.

The order of read_columns decides the output field order after subtable_name. If the result is written to a TDengine sink, keep normal columns before TAGS columns so the sink can split column values and TAGS values correctly.

common options​

Source plugin common parameters, please refer to Source Common Options for details.

Output Schema​

The output table always starts with the reserved subtable_name column, which carries the TDengine sub table name each row came from. Subsequent columns are the columns declared in read_columns (or all columns when read_columns is unset), in the order specified by read_columns. TAGS columns follow the normal columns in the same order you list them.

Examples​

Read all sub tables in a time range​

env {
  parallelism = 2
  job.mode = "BATCH"
}

source {
  TDengine {
    url = "jdbc:TAOS-RS://localhost:6041/"
    username = "root"
    password = "taosdata"
    database = "power"
    stable = "meters"
    lower_bound = "2018-10-03 14:38:05.000"
    upper_bound = "2018-10-03 14:38:16.801"
    plugin_output = "tdengine_result"
  }
}

Read selected sub tables and columns​

source {
  TDengine {
    url = "jdbc:TAOS-RS://localhost:6041/"
    username = "root"
    password = "taosdata"
    database = "power"
    stable = "meters"
    lower_bound = "2018-10-03 14:38:05.000"
    upper_bound = "2018-10-03 14:38:16.801"
    sub_tables = ["d1001", "d1002"]
    read_columns = ["ts", "current", "voltage", "phase", "off", "nc", "location", "groupid"]
  }
}

This reads only sub tables d1001 and d1002 under super table meters. The TAGS columns (location, groupid) are placed at the end of read_columns so that a downstream TDengine sink can correctly split normal columns and TAGS values.

Read from TDengine and write to TDengine​

env {
  parallelism = 2
  job.mode = "BATCH"
}

source {
  TDengine {
    url = "jdbc:TAOS-RS://tdengine-src:6041/"
    username = "root"
    password = "taosdata"
    database = "power"
    stable = "meters"
    lower_bound = "2018-10-03 14:38:05.000"
    upper_bound = "2018-10-03 14:38:16.801"
    plugin_output = "tdengine_result"
  }
}

sink {
  TDengine {
    url = "jdbc:TAOS-RS://tdengine-sink:6041/"
    username = "root"
    password = "taosdata"
    database = "power2"
    stable = "meters2"
    timezone = "UTC"
  }
}

The sink relies on subtable_name from the source as the target sub table name when writing back. The number of TAGS values is read from the target super table metadata, so the target super table must already exist.

Changelog​

Change Log
ChangeCommitVersion
[Feature][connector-tdengine] Support subtable and fieldNames in tdengine source (#9593)https://github.com/apache/seatunnel/commit/b136a0dc432.3.12
[improve] tdengine options (#9399)https://github.com/apache/seatunnel/commit/ff122fe4052.3.12
[Feature][Connector-V2] Support multi-table sink feature for TDengine (#9215)https://github.com/apache/seatunnel/commit/98b593f0952.3.11
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Fix][Connector-V2] Fix NullPointerException when column or tag contains null value in TDengine sink (#9158)https://github.com/apache/seatunnel/commit/a047cab5462.3.11
[Fix][Connector][TDEngine] TDEngine support NCHAR type (#8411)https://github.com/apache/seatunnel/commit/88c92ae1b12.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Improve][Connector-V2] Close all ResultSet after used (#7389)https://github.com/apache/seatunnel/commit/853e9732122.3.8
[Fix][Connector-tdengine] Fix sql exception and concurrentmodifyexception when connect to taos and read datahttps://github.com/apache/seatunnel/commit/a18fca80062.3.7
[Bugfix][TDengine] Fix the issue of losing the driver due to multiple calls to the submit job REST API #6581 (#6596)https://github.com/apache/seatunnel/commit/470bb974342.3.5
[improve][connector-tdengine] support read bool column from tdengine (#6025)https://github.com/apache/seatunnel/commit/af39235ee32.3.4
[Bugfix][TDengine] Fix the degree of multiple parallelism affects driver loading (#6020)https://github.com/apache/seatunnel/commit/b6ebbd47b22.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)https://github.com/apache/seatunnel/commit/51c0d709ba2.3.4
[Hotfix][Connector] Fixed TDengine connector using jdbc driver to cause loading error (#4598)https://github.com/apache/seatunnel/commit/78f7989b812.3.2
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector-V2] add tdengine source (#2832)https://github.com/apache/seatunnel/commit/acf4d5b1b42.3.1