Skip to main content
Version: Next

AmazonDynamoDB

Amazon DynamoDB sink connector

Description​

The Amazon DynamoDB sink connector writes SeaTunnel rows to a DynamoDB table.

The target table must already exist. The connector writes each row as a DynamoDB item and uses batch write requests. It supports single-table writes and multi-table writes when the upstream row carries a table id.

Supported Engines​

Spark
Flink
SeaTunnel Zeta

Key Features​

Options​

nametyperequireddefault valuedescription
urlstringyes-DynamoDB endpoint URL.
regionstringyes-AWS region of the DynamoDB service.
access_key_idstringyes-AWS access key ID.
secret_access_keystringyes-AWS secret access key.
tablestringyes-DynamoDB table name to write to.
batch_sizeintno25Records buffered for one batch write request.
multi_table_sink_replicaintno-Sink writer replicas for each table.
max_retriesintno10Retries for unprocessed items.
retry_base_delay_mslongno100Initial retry backoff delay in milliseconds.
retry_max_delay_mslongno5000Maximum retry backoff delay in milliseconds.
common-optionsobjectno-Sink plugin common parameters.

url [string]​

The DynamoDB endpoint URL, for example https://dynamodb.us-east-1.amazonaws.com.

When testing with DynamoDB Local, use the local endpoint, for example http://127.0.0.1:8000.

region [string]​

The AWS region of the DynamoDB service, such as us-east-1.

access_key_id [string]​

The AWS access key ID used to connect to DynamoDB.

secret_access_key [string]​

The AWS secret access key used to connect to DynamoDB.

table [string]​

The DynamoDB table name to write to.

For a normal single-table job, set this to the target table name. In a multi-table pipeline, the writer uses the table id carried by each row as the target table name and falls back to this configured table when the row has no table id.

batch_size [int]​

The number of records buffered for one DynamoDB batch write request.

DynamoDB batch write supports up to 25 write requests per call, so the default is 25. Do not set this higher than 25; larger values do not match the DynamoDB batch write API limit.

multi_table_sink_replica [int]​

Optional common sink option used by multi-table sink jobs. For details, see Sink Common Options.

max_retries [int]​

The maximum number of retries when DynamoDB returns unprocessed items from a batch write request.

retry_base_delay_ms [long]​

The base delay, in milliseconds, used by exponential backoff between retries.

retry_max_delay_ms [long]​

The maximum delay, in milliseconds, between retries.

common options​

Sink plugin common parameters, please refer to Sink Common Options for details.

Usage Notes​

  • Create the target DynamoDB table before starting the SeaTunnel job. The sink does not create tables or key schemas.
  • access_key_id and secret_access_key are required by this connector. For DynamoDB Local, use dummy values accepted by the local service.
  • DynamoDB accepts at most 25 write requests in one batch write call, so keep batch_size at or below 25.
  • The sink retries unprocessed items with exponential backoff. It does not provide exactly-once guarantees.

Data Type Mapping​

SeaTunnel Data TypeDynamoDB Attribute Type
BOOLEANBOOL
TINYINTN
SMALLINTN
INTN
BIGINTN
FLOATN
DOUBLEN
DECIMALN
STRINGS
TIMES
DATES
TIMESTAMPS
BYTESB
MAPM
ARRAYL
NULLNULL

Task Example​

The following example reads rows from source_table and writes them to sink_table.

env {
  parallelism = 2
  job.mode = "BATCH"
}

source {
  AmazonDynamoDB {
    url = "http://127.0.0.1:8000"
    region = "us-east-1"
    access_key_id = "dummy-key"
    secret_access_key = "dummy-secret"
    table = "source_table"
    parallelism = 2
    schema = {
      fields {
        id = string
        c_map = "map<string, smallint>"
        c_array = "array<tinyint>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_decimal = "decimal(2, 1)"
        c_bytes = bytes
        c_date = date
        c_timestamp = timestamp
      }
    }
  }
}

sink {
  AmazonDynamoDB {
    url = "http://127.0.0.1:8000"
    region = "us-east-1"
    access_key_id = "dummy-key"
    secret_access_key = "dummy-secret"
    table = "sink_table"
    batch_size = 25
    max_retries = 10
    retry_base_delay_ms = 100
    retry_max_delay_ms = 5000
  }
}

Changelog​

Change Log
ChangeCommitVersion
[Improve][Core] Unify the aws-sdk-v2 version to 2.31.30 (#9698)https://github.com/apache/seatunnel/commit/41c251cc8a2.3.12
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Fix] Fix error log name for SourceSplitEnumerator implements class (#8817)https://github.com/apache/seatunnel/commit/55ed90ecaf2.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[improve] update amazondynamodb connector (#8601)https://github.com/apache/seatunnel/commit/a69efca0fd2.3.10
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Feature][Doris] Add Doris type converter (#6354)https://github.com/apache/seatunnel/commit/51899918432.3.6
[Feature][Core] Upgrade flink source translation (#5100)https://github.com/apache/seatunnel/commit/5aabb14a942.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Code clean for AmazonDynamoDB connector (#5791)https://github.com/apache/seatunnel/commit/a17dd7afc12.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[connector-v2] add amazondynamicdb source split (#5275)https://github.com/apache/seatunnel/commit/740c14422d2.3.4
Support config column/primaryKey/constraintKey in schema (#5564)https://github.com/apache/seatunnel/commit/eac76b4e502.3.4
[Improve]Remove scheduler in Dynamodb sink (#5248)https://github.com/apache/seatunnel/commit/9e033a824e2.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)https://github.com/apache/seatunnel/commit/51c0d709ba2.3.4
[Bugfix][AmazonDynamoDB] Fix the problem that all table data cannot be obtained (#5146)https://github.com/apache/seatunnel/commit/09995159a02.3.3
Merge branch 'dev' into merge/cdchttps://github.com/apache/seatunnel/commit/4324ee19122.3.1
[Improve][Project] Code format with spotless plugin.https://github.com/apache/seatunnel/commit/423b5830382.3.1
[improve][api] Refactoring schema parse (#4157)https://github.com/apache/seatunnel/commit/b2f573a13e2.3.1
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector] add get source method to all source connector (#3846)https://github.com/apache/seatunnel/commit/417178fb842.3.1
[Feature][API &amp; Connector &amp; Doc] add parallelism and column projection interface (#3829)https://github.com/apache/seatunnel/commit/b9164b8ba12.3.1
[Hotfix][API] env required option can not set default value (#3584)https://github.com/apache/seatunnel/commit/c5a23024f62.3.0
[Feature][Connector-V2][AmazonDynamoDB] Add Factory for AmazonDynamoDB (#3348)https://github.com/apache/seatunnel/commit/a0068efdbf2.3.0
[Improve][Connector-V2][AmazonDynamoDB] Unified exception for AmazonDynamoDB source & sink connector (#3333)https://github.com/apache/seatunnel/commit/17bc5adcef2.3.0
[Connector-V2][Chore] Canonical name for AmazonDynamodb (#3321)https://github.com/apache/seatunnel/commit/e216eb9a6b2.3.0
[Feature][Connector-V2] [Amazondynamodb Connector]add amazondynamodb source & sink connnector (#3166)https://github.com/apache/seatunnel/commit/183bac02f02.3.0