跳到主要内容
版本:3.1.0

AmazonDynamoDB

Amazon DynamoDB 写入连接器

描述​

Amazon DynamoDB 写入连接器用于将 SeaTunnel 数据行写入 DynamoDB 表。

目标表必须提前创建。连接器会把每一行写成一个 DynamoDB item,并使用批量写入请求。它支持单表写入,也支持上游数据行携带表名时的多表写入。

支持的引擎​

Spark
Flink
SeaTunnel Zeta

主要特性​

选项​

名称类型必填默认值说明
urlstring是-DynamoDB 服务地址。
regionstring是-DynamoDB 所在的 AWS 区域。
access_key_idstring是-AWS access key ID。
secret_access_keystring是-AWS secret access key。
tablestring是-要写入的 DynamoDB 表名。
batch_sizeint否25一次批量写入请求缓存的记录数。
multi_table_sink_replicaint否1每张表对应的 Sink Writer 副本数。
max_retriesint否10未处理 item 的最大重试次数,必须大于等于 0。
retry_base_delay_mslong否100初始重试等待时间,单位毫秒。
retry_max_delay_mslong否5000最大重试等待时间,单位毫秒。
common-optionsobject否-Sink 插件通用参数。

url [string]​

DynamoDB 服务地址,例如 https://dynamodb.us-east-1.amazonaws.com。

如果使用 DynamoDB Local 测试,可以填写本地地址,例如 http://127.0.0.1:8000。

region [string]​

DynamoDB 所在的 AWS 区域,例如 us-east-1。

access_key_id [string]​

连接 DynamoDB 使用的 AWS access key ID。

secret_access_key [string]​

连接 DynamoDB 使用的 AWS secret access key。

table [string]​

要写入的 DynamoDB 表名。

普通单表任务中,这里填写目标表名。多表任务中,写入器会优先使用每条数据自带的表名作为目标表;如果数据没有携带表名,则回退使用这里配置的表名。

batch_size [int]​

一次 DynamoDB 批量写入请求缓存的记录数。

DynamoDB batch write 每次最多支持 25 条写请求,所以默认值为 25。 不要把该值设置为大于 25,否则会超过 DynamoDB batch write API 的限制。

multi_table_sink_replica [int]​

多表写入任务可使用的 Sink 通用选项。更多说明请参考 Sink 通用选项。

max_retries [int]​

当 DynamoDB 在批量写入结果中返回未处理 item 时,最多重试的次数。 设置为 0 表示不重试,该值不能为负数。

retry_base_delay_ms [long]​

重试之间指数退避的基础等待时间,单位为毫秒。

retry_max_delay_ms [long]​

重试之间的最大等待时间,单位为毫秒。

通用选项​

Sink 连接器通用参数,请参考 Sink 通用选项。

使用说明​

  • 启动 SeaTunnel 任务前,需要先创建目标 DynamoDB 表。该写入连接器不会自动创建表或主键结构。
  • access_key_id 和 secret_access_key 是必填项。使用 DynamoDB Local 时,可以填写本地服务接受的占位值。
  • DynamoDB 每次 batch write 最多接受 25 条写请求,所以 batch_size 不要超过 25。
  • 写入连接器会对未处理 item 做指数退避重试,但不提供精确一次保证。

数据类型映射​

SeaTunnel 数据类型DynamoDB 属性类型
BOOLEANBOOL
TINYINTN
SMALLINTN
INTN
BIGINTN
FLOATN
DOUBLEN
DECIMALN
STRINGS
TIMES
DATES
TIMESTAMPS
BYTESB
MAPM
ARRAYL
NULLNULL

任务示例​

下面的示例从 source_table 读取数据,并写入 sink_table。

env {
parallelism = 2
job.mode = "BATCH"
}

source {
AmazonDynamoDB {
url = "http://127.0.0.1:8000"
region = "us-east-1"
access_key_id = "dummy-key"
secret_access_key = "dummy-secret"
table = "source_table"
parallelism = 2
schema = {
fields {
id = string
c_map = "map<string, smallint>"
c_array = "array<tinyint>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_decimal = "decimal(2, 1)"
c_bytes = bytes
c_date = date
c_timestamp = timestamp
}
}
}
}

sink {
AmazonDynamoDB {
url = "http://127.0.0.1:8000"
region = "us-east-1"
access_key_id = "dummy-key"
secret_access_key = "dummy-secret"
table = "sink_table"
batch_size = 25
max_retries = 10
retry_base_delay_ms = 100
retry_max_delay_ms = 5000
}
}

变更日志​

Change Log
ChangeCommitVersion
[Feature][Connector-V2][AmazonDynamoDB] Support multi-table source reads (#12665)https://github.com/apache/seatunnel/commit/0c20c86ef83.1.0
[Improve][Connector-V2][AmazonDynamoDB] Migrate max retries validation to OptionRule (#11821)https://github.com/apache/seatunnel/commit/03ef9f1c003.0.0
[Improve][Connector-V2] Improve source split round-robin assignment for Easysearch, AmazonDynamoDB, TiDB CDC and Typesense (#11607)https://github.com/apache/seatunnel/commit/9487cc8e163.0.0
[Feature][Connector-V2] Add multi-table sink support for AmazonDynamo… (#10497)https://github.com/apache/seatunnel/commit/0b1c9a52c93.0.0
[Improve][Core] Unify the aws-sdk-v2 version to 2.31.30 (#9698)https://github.com/apache/seatunnel/commit/41c251cc8a2.3.12
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Fix] Fix error log name for SourceSplitEnumerator implements class (#8817)https://github.com/apache/seatunnel/commit/55ed90ecaf2.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[improve] update amazondynamodb connector (#8601)https://github.com/apache/seatunnel/commit/a69efca0fd2.3.10
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Feature][Doris] Add Doris type converter (#6354)https://github.com/apache/seatunnel/commit/51899918432.3.6
[Feature][Core] Upgrade flink source translation (#5100)https://github.com/apache/seatunnel/commit/5aabb14a942.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Code clean for AmazonDynamoDB connector (#5791)https://github.com/apache/seatunnel/commit/a17dd7afc12.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[connector-v2] add amazondynamicdb source split (#5275)https://github.com/apache/seatunnel/commit/740c14422d2.3.4
Support config column/primaryKey/constraintKey in schema (#5564)https://github.com/apache/seatunnel/commit/eac76b4e502.3.4
[Improve]Remove scheduler in Dynamodb sink (#5248)https://github.com/apache/seatunnel/commit/9e033a824e2.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)https://github.com/apache/seatunnel/commit/51c0d709ba2.3.4
[Bugfix][AmazonDynamoDB] Fix the problem that all table data cannot be obtained (#5146)https://github.com/apache/seatunnel/commit/09995159a02.3.3
Merge branch 'dev' into merge/cdchttps://github.com/apache/seatunnel/commit/4324ee19122.3.1
[Improve][Project] Code format with spotless plugin.https://github.com/apache/seatunnel/commit/423b5830382.3.1
[improve][api] Refactoring schema parse (#4157)https://github.com/apache/seatunnel/commit/b2f573a13e2.3.1
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector] add get source method to all source connector (#3846)https://github.com/apache/seatunnel/commit/417178fb842.3.1
[Feature][API &amp; Connector &amp; Doc] add parallelism and column projection interface (#3829)https://github.com/apache/seatunnel/commit/b9164b8ba12.3.1
[Hotfix][API] env required option can not set default value (#3584)https://github.com/apache/seatunnel/commit/c5a23024f62.3.0
[Feature][Connector-V2][AmazonDynamoDB] Add Factory for AmazonDynamoDB (#3348)https://github.com/apache/seatunnel/commit/a0068efdbf2.3.0
[Improve][Connector-V2][AmazonDynamoDB] Unified exception for AmazonDynamoDB source & sink connector (#3333)https://github.com/apache/seatunnel/commit/17bc5adcef2.3.0
[Connector-V2][Chore] Canonical name for AmazonDynamodb (#3321)https://github.com/apache/seatunnel/commit/e216eb9a6b2.3.0
[Feature][Connector-V2] [Amazondynamodb Connector]add amazondynamodb source & sink connnector (#3166)https://github.com/apache/seatunnel/commit/183bac02f02.3.0