Skip to main content
Version: 2.3.1

SelectDB Cloud

SelectDB Cloud sink connector

Description

Used to send data to SelectDB Cloud. Both support streaming and batch mode. The internal implementation of SelectDB Cloud sink connector upload after batch caching and commit the CopyInto sql to load data into the table.

tip

Version Supported

  • supported SelectDB Cloud version is >= 2.2.x

Key features

Options

nametyperequireddefault value
load-urlstringyes-
jdbc-urlstringyes-
cluster-namestringyes-
usernamestringyes-
passwordstringyes-
table.identifierstringyes-
sink.enable-deleteboolnofalse
selectdb.configmapyes-
sink.buffer-sizeintno10 1024 1024 (1MB)
sink.buffer-countintno10000
sink.max-retriesintno3

load-url [string]

SelectDB Cloud warehouse http address, the format is warehouse_ip:http_port

jdbc-url [string]

SelectDB Cloud warehouse jdbc address, the format is warehouse_ip:mysql_port

cluster-name [string]

SelectDB Cloud cluster name

username [string]

SelectDB Cloud user username

password [string]

SelectDB Cloud user password

table.identifier [string]

The name of SelectDB Cloud table, the format is database.table

sink.enable-delete [string]

Whether to enable deletion. This option requires SelectDB Cloud table to enable batch delete function, and only supports Unique model.

ALTER TABLE example_db.my_table ENABLE FEATURE "BATCH_DELETE";

selectdb.config [map]

Write property configuration

CSV Write:

selectdb.config {
file.type="csv"
file.column_separator=","
file.line_delimiter="\n"
}

JSON Write:

selectdb.config {
file.type="json"
}

sink.buffer-size [string]

The maximum capacity of the cache, in bytes, that is flushed to the object storage. The default is 10MB. it is not recommended to modify it.

sink.buffer-count [string]

Maximum number of entries flushed to the object store. The default value is 10000. it is not recommended to modify.

sink.max-retries [string]

The maximum number of retries in the Commit phase, the default is 3.

Example

Use JSON format to import data

sink {
SelectDBCloud {
load-url="warehouse_ip:http_port"
jdbc-url="warehouse_ip:mysql_port"
cluster-name="Cluster"
table.identifier="test.test"
username="admin"
password="******"
selectdb.config {
file.type="json"
}
}
}

Use CSV format to import data

sink {
SelectDBCloud {
load-url="warehouse_ip:http_port"
jdbc-url="warehouse_ip:mysql_port"
cluster-name="Cluster"
table.identifier="test.test"
username="admin"
password="******"
selectdb.config {
file.type="csv"
file.column_separator=","
file.line_delimiter="\n"
}
}
}

Changelog

next version

  • [Feature] Support SelectDB Cloud Sink Connector 3958
  • [Improve] Refactor some SelectDB Cloud Sink code as well as support copy into batch and async flush and cdc 4312