Skip to main content
Version: Next

Databend

Databend source connector

Support Those Engines​

Spark
Flink
SeaTunnel Zeta

Key Features​

Description​

A source connector for reading data from Databend using the Databend JDBC driver. You can read a single table with database + table, run an ad-hoc query with query, or supply an explicit statement through sql. The connector executes the query in batch mode and returns each result row as a SeaTunnel row.

The connector supports column projection through standard SQL and exposes a few JDBC tuning options such as fetch_size and ssl. It does not currently read multiple tables in a single source block; configure one Databend source per table.

Dependencies​

  1. You need to download the Databend JDBC driver jar package and add it to the directory ${SEATUNNEL_HOME}/plugins/.

For SeaTunnel Zeta​

  1. You need to download the Databend JDBC driver jar package and add it to the directory ${SEATUNNEL_HOME}/lib/.

Supported DataSource Info​

In order to use the Databend connector, the following dependencies are required. They can be downloaded via install-plugin.sh or from the Maven central repository.

DatasourceSupported VersionsDependency
Databend1.2.x and aboveDownload

Data Type Mapping​

Databend Data TypeSeaTunnel Data Type
BOOLEANBOOLEAN
TINYINTTINYINT
SMALLINTSMALLINT
INTINT
BIGINTBIGINT
FLOATFLOAT
DOUBLEDOUBLE
DECIMALDECIMAL
STRINGSTRING
VARCHARSTRING
CHARSTRING
TIMESTAMPTIMESTAMP
DATEDATE
TIMETIME
BINARYBYTES

Source Options​

Basic Configuration:

NameTypeRequiredDefault ValueDescription
urlStringYes-Databend JDBC connection URL. It must start with jdbc:databend://
usernameStringYes-Databend database username
passwordStringYes-Databend database password
databaseStringNo-Databend database name, defaults to the database name specified in the connection URL
tableStringNo-Databend table name
queryStringNo-Databend query statement. If set, it overrides database and table settings
sqlStringNo-Alias-style custom SQL statement. If both sql and query are set, sql takes precedence
fetch_sizeIntegerNo1Number of records to fetch from Databend at once. Set it higher for large reads. Set to 0 to use the JDBC driver default
sslBooleanNofalseWhether to use SSL for the Databend connection
jdbc_configMapNo-Additional JDBC connection configuration, such as load balancing strategies
common-optionsNo-Source plugin common parameters, please refer to Source Common Options for details.

You must configure either sql, query, or both database and table. When more than one of them is configured, the read SQL is chosen in this order: sql, then query, then SELECT * FROM database.table. The connector does not currently support table_list, so configure one Databend source block for each table.

Task Examples​

Single Table Reading​

env {
  parallelism = 2
  job.mode = "BATCH"
}

source {
  Databend {
    url = "jdbc:databend://localhost:8000"
    username = "root"
    password = ""
    database = "default"
    table = "users"
  }
}

sink {
  Console {}
}

Using Custom Query​

source {
  Databend {
    url = "jdbc:databend://localhost:8000"
    username = "root"
    password = ""
    query = "SELECT id, name, age FROM default.users WHERE age > 18"
  }
}

Using SSL​

source {
  Databend {
    url = "jdbc:databend://databend.example.com:8000/default"
    username = "root"
    password = ""
    sql = "SELECT * FROM default.users"
    ssl = true
    fetch_size = 1000
  }
}

Filter On A Computed Column​

Use any expression supported by Databend in query to project and filter rows before they reach SeaTunnel:

source {
  Databend {
    url = "jdbc:databend://localhost:8000"
    username = "root"
    password = ""
    query = "SELECT id, name, age FROM default.users WHERE age >= 18 AND starts_with(name, 'A') ORDER BY id"
  }
}

Changelog​

Change Log
ChangeCommitVersion
[Feature][Connector-V2] Support databend source/sink connector (#9331)https://github.com/apache/seatunnel/commit/2f96f2e46c2.3.12