OpenMldb

OpenMldb source connector

Support Those Engines

Spark
Flink
SeaTunnel Zeta

Description

Used to read data from OpenMLDB. The connector executes the configured SQL statement against OpenMLDB and turns the result rows into SeaTunnel records. Both standalone and cluster deployment modes are supported.

Key features

• batch
• stream
• exactly-once
• column projection
• parallelism
• support user-defined split

Data Type Mapping

OpenMLDB types are mapped to SeaTunnel types according to the result schema of the configured sql statement. Columns whose types are not natively understood by SeaTunnel will cause the read to fail with an UNSUPPORTED_DATA_TYPE error.

OpenMLDB Data Type	SeaTunnel Data Type
bool	boolean
smallint	smallint
int	int
bigint	bigint
float / double	float / double
string / varchar	string
date	date
timestamp	timestamp

Source Options

name	type	required	default value	description
cluster_mode	boolean	yes	-	Whether to connect to OpenMLDB in cluster mode. Set to false for standalone mode.
sql	string	yes	-	SQL statement to execute against OpenMLDB. Column names and types follow the result.
database	string	yes	-	The OpenMLDB database name to connect to.
host	string	no	-	Required when cluster_mode is false. Host of the standalone OpenMLDB server.
port	int	no	-	Required when cluster_mode is false. Port of the standalone OpenMLDB server.
zk_host	string	no	-	Required when cluster_mode is true. ZooKeeper host list of the OpenMLDB cluster.
zk_path	string	no	-	Required when cluster_mode is true. ZooKeeper path of the OpenMLDB cluster.
session_timeout	int	no	10000	OpenMLDB session timeout in milliseconds.
request_timeout	int	no	60000	OpenMLDB request timeout in milliseconds.
common-options		no	-	Source plugin common parameters, please refer to Source Common Options for details.

cluster_mode [boolean]

Whether to connect to OpenMLDB in cluster mode. When it is false, configure host and port. When it is true, configure zk_host and zk_path.

sql [string]

The SQL statement to execute against OpenMLDB. The result set columns become the schema of the emitted SeaTunnel rows.

database [string]

The OpenMLDB database name to connect to. The configured database must exist on the target OpenMLDB instance.

host [string]

OpenMLDB host. Only used when cluster_mode is false (standalone mode).

port [int]

OpenMLDB port. Only used when cluster_mode is false (standalone mode).

zk_host [string]

ZooKeeper host list for the OpenMLDB cluster, for example zk-1:2181,zk-2:2181,zk-3:2181. Only used when cluster_mode is true.

zk_path [string]

ZooKeeper path of the OpenMLDB cluster, for example /openmldb. Only used when cluster_mode is true.

session_timeout [int]

OpenMLDB session timeout in milliseconds. Defaults to 10000 (10 seconds).

request_timeout [int]

OpenMLDB request timeout in milliseconds. Defaults to 60000 (60 seconds).

common options

Source plugin common parameters, please refer to Source Common Options for details.

Task Example

Standalone mode

source {
  OpenMldb {
    host = "172.17.0.2"
    port = 6527
    sql = "select * from demo_table1"
    database = "demo_db"
    cluster_mode = false
  }
}

Cluster mode

source {
  OpenMldb {
    zk_host = "zk-1:2181,zk-2:2181,zk-3:2181"
    zk_path = "/openmldb"
    sql = "select * from demo_table1"
    database = "demo_db"
    cluster_mode = true
  }
}

With downstream sink

A typical end-to-end job that reads from OpenMLDB and prints the rows through the Console sink.

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  OpenMldb {
    host = "172.17.0.2"
    port = 6527
    sql = "select id, name from demo_table1"
    database = "demo_db"
    cluster_mode = false
  }
}

sink {
  Console {
  }
}

Changelog

Change Log

Change	Commit	Version
[improve] openmldb options (#9166)	https://github.com/apache/seatunnel/commit/d324fc59a4	2.3.11
[Improve][Common] Introduce new error define rule (#5793)	https://github.com/apache/seatunnel/commit/9d1b2582b2	2.3.4
[Improve][build] Give the maven module a human readable name (#4114)	https://github.com/apache/seatunnel/commit/d7cd601051	2.3.1
[Improve][Project] Code format with spotless plugin. (#4101)	https://github.com/apache/seatunnel/commit/a2ab166561	2.3.1
[Feature][Connector] add get source method to all source connector (#3846)	https://github.com/apache/seatunnel/commit/417178fb84	2.3.1
[Feature][API & Connector & Doc] add parallelism and column projection interface (#3829)	https://github.com/apache/seatunnel/commit/b9164b8ba1	2.3.1
[Hotfix][OptionRule] Fix option rule about all connectors (#3592)	https://github.com/apache/seatunnel/commit/226dc6a119	2.3.0
[Feature][Conenctor-V2] Add openmldb source connector (#3313)	https://github.com/apache/seatunnel/commit/e68ecf7bef	2.3.0