Skip to main content
Version: Next

InfluxDB

InfluxDB source connector

Description​

Read external data source data through InfluxDB.

Key features​

supports query SQL and can achieve projection effect.

Options​

nametyperequireddefault value
urlstringyes-
sqlstringyes-
schemaconfigyes-
databasestringyes
usernamestringno-
passwordstringno-
lower_boundlongno-
upper_boundlongno-
partition_numintno-
split_columnstringno-
epochstringnon
connect_timeout_mslongno15000
query_timeout_secintno3
common-optionsconfigno-

url​

the url to connect to influxDB e.g.

http://influxdb-host:8086

sql [string]​

The query sql used to search data

select name,age from test

schema [config]​

fields [Config]​

The schema information of upstream data. e.g.

schema {
    fields {
        name = string
        age = int
    }
  }

database [string]​

The influxDB database

username [string]​

the username of the influxDB when you select

password [string]​

the password of the influxDB when you select

split_column [string]​

the split_column of the influxDB when you select

Tips:

  • influxDB tags is not supported as a segmented primary key because the type of tags can only be a string
  • influxDB time is not supported as a segmented primary key because the time field cannot participate in mathematical calculation
  • Currently, split_column only supports integer data segmentation, and does not support float, string, date and other types.

upper_bound [long]​

upper bound of the split_columncolumn

lower_bound [long]​

lower bound of the split_column column

     split the $split_column range into $partition_num parts
     if partition_num is 1, use the whole `split_column` range
     if partition_num < (upper_bound - lower_bound), use (upper_bound - lower_bound) partitions
     
     eg: lower_bound = 1, upper_bound = 10, partition_num = 2
     sql = "select * from test where age > 0 and age < 10"
     
     split result

     split 1: select * from test where ($split_column >= 1 and $split_column < 6)  and (  age > 0 and age < 10 )
     
     split 2: select * from test where ($split_column >= 6 and $split_column < 11) and (  age > 0 and age < 10 )

partition_num [int]​

the partition_num of the InfluxDB when you select

Tips: Ensure that upper_bound minus lower_bound is divided bypartition_num, otherwise the query results will overlap

epoch [string]​

returned time precision

  • Optional values: H, m, s, MS, u, n
  • default value: n

query_timeout_sec [int]​

the query_timeout of the InfluxDB when you select, in seconds

connect_timeout_ms [long]​

the timeout for connecting to InfluxDB, in milliseconds

common options​

Source plugin common parameters, please refer to Source Common Options for details

Examples​

Example of multi parallelism and multi partition scanning

source {

    InfluxDB {
        url = "http://influxdb-host:8086"
        sql = "select label, value, rt, time from test"
        database = "test"
        upper_bound = 100
        lower_bound = 1
        partition_num = 4
        split_column = "value"
        schema {
            fields {
                label = STRING
                value = INT
                rt = STRING
                time = BIGINT
            }
    }

}

Example of not using partition scan

source {

    InfluxDB {
        url = "http://influxdb-host:8086"
        sql = "select label, value, rt, time from test"
        database = "test"
        schema {
            fields {
                label = STRING
                value = INT
                rt = STRING
                time = BIGINT
            }
    }

}

Changelog​

Change Log
ChangeCommitVersion
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eebdev
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Improve] Improve some connectors prepare check error message (#7465)https://github.com/apache/seatunnel/commit/6930a25edd2.3.8
[Improve][Connector] Add multi-table sink option check (#7360)https://github.com/apache/seatunnel/commit/2489f6446b2.3.7
[Feature][Core] Support using upstream table placeholders in sink options and auto replacement (#7131)https://github.com/apache/seatunnel/commit/c4ca74122c2.3.6
Support multi-table sink feature for influxdb (#6278)https://github.com/apache/seatunnel/commit/56f13e920d2.3.5
[Improve][Zeta] Add classloader cache mode to fix metaspace leak (#6355)https://github.com/apache/seatunnel/commit/9c3c2f183d2.3.5
[Test][E2E] Add thread leak check for connector (#5773)https://github.com/apache/seatunnel/commit/1f2f3fc5f02.3.4
[BugFix][InfluxDBSource] Resolve invalid SQL in initColumnsIndex method caused by direct QUERY_LIMIT appendage with 'tz' function. (#4829)https://github.com/apache/seatunnel/commit/deed9c62c32.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
Support config column/primaryKey/constraintKey in schema (#5564)https://github.com/apache/seatunnel/commit/eac76b4e502.3.4
[Improve][Connector-V2] Remove scheduler in InfluxDB sink (#5271)https://github.com/apache/seatunnel/commit/f459f500cb2.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)https://github.com/apache/seatunnel/commit/51c0d709ba2.3.4
Merge branch 'dev' into merge/cdchttps://github.com/apache/seatunnel/commit/4324ee19122.3.1
[Improve][Project] Code format with spotless plugin.https://github.com/apache/seatunnel/commit/423b5830382.3.1
[improve][api] Refactoring schema parse (#4157)https://github.com/apache/seatunnel/commit/b2f573a13e2.3.1
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Improve][SourceConnector] Unifie InfluxDB source fields to schema (#3897)https://github.com/apache/seatunnel/commit/85a984a64f2.3.1
[Feature][Connector] add get source method to all source connector (#3846)https://github.com/apache/seatunnel/commit/417178fb842.3.1
[Feature][API &amp; Connector &amp; Doc] add parallelism and column projection interface (#3829)https://github.com/apache/seatunnel/commit/b9164b8ba12.3.1
[Hotfix][OptionRule] Fix option rule about all connectors (#3592)https://github.com/apache/seatunnel/commit/226dc6a1192.3.0
[Improve][Connector-V2][Influxdb] Unified exception for influxdb source & sink connector (#3558)https://github.com/apache/seatunnel/commit/4686f35d682.3.0
[Feature][Connector][influx] Expose configurable options in influx db (#3392)https://github.com/apache/seatunnel/commit/b247ff0aef2.3.0
[Feature][Connector-V2] influxdb sink connector (#3174)https://github.com/apache/seatunnel/commit/630e8847912.3.0
[Feature][Connector-V2] Add influxDB connector source (#2697)https://github.com/apache/seatunnel/commit/1d70ea30842.3.0-beta