AmazonDynamoDB
AmazonDynamoDB source connector
Description
Read data from Amazon DynamoDB.
Key features
Options
name | type | required | default value |
---|---|---|---|
url | string | yes | - |
region | string | yes | - |
access_key_id | string | yes | - |
secret_access_key | string | yes | - |
table | string | yes | - |
schema | config | yes | - |
common-options | yes | - | |
scan_item_limit | false | - | |
parallel_scan_threads | false | - |
url [string]
The URL to read to Amazon Dynamodb.
region [string]
The region of Amazon Dynamodb.
accessKeyId [string]
The access id of Amazon DynamoDB.
secretAccessKey [string]
The access secret of Amazon DynamoDB.
table [string]
The table of Amazon DynamoDB.
schema [Config]
fields [config]
Amazon Dynamodb is a NOSQL database service of support keys-value storage and document data structure,there is no way to get the data type.Therefore, we must configure schema.
such as:
schema {
fields {
id = int
key_aa = string
key_bb = string
}
}
common options
Source Plugin common parameters, refer to Source Plugin for details
scan_item_limit
number of item each scan request should return
parallel_scan_threads
number of logical segments for parallel scan
Example
Amazondynamodb {
url = "http://127.0.0.1:8000"
region = "us-east-1"
accessKeyId = "dummy-key"
secretAccessKey = "dummy-secret"
table = "TableName"
schema = {
fields {
artist = string
c_map = "map<string, array<int>>"
c_array = "array<int>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_decimal = "decimal(30, 8)"
c_null = "null"
c_bytes = bytes
c_date = date
c_timestamp = timestamp
}
}
}
Changelog
Change Log
Change | Commit | Version |
---|---|---|
[Fix] Fix error log name for SourceSplitEnumerator implements class (#8817) | https://github.com/apache/seatunnel/commit/55ed90ecaf | dev |
[Improve] restruct connector common options (#8634) | https://github.com/apache/seatunnel/commit/f3499a6eeb | dev |
[improve] update amazondynamodb connector (#8601) | https://github.com/apache/seatunnel/commit/a69efca0fd | dev |
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03c | 2.3.9 |
[Feature][Doris] Add Doris type converter (#6354) | https://github.com/apache/seatunnel/commit/5189991843 | 2.3.6 |
[Feature][Core] Upgrade flink source translation (#5100) | https://github.com/apache/seatunnel/commit/5aabb14a94 | 2.3.4 |
[Improve][Common] Introduce new error define rule (#5793) | https://github.com/apache/seatunnel/commit/9d1b2582b2 | 2.3.4 |
[Improve] Code clean for AmazonDynamoDB connector (#5791) | https://github.com/apache/seatunnel/commit/a17dd7afc1 | 2.3.4 |
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755) | https://github.com/apache/seatunnel/commit/8de7408100 | 2.3.4 |
[connector-v2] add amazondynamicdb source split (#5275) | https://github.com/apache/seatunnel/commit/740c14422d | 2.3.4 |
Support config column/primaryKey/constraintKey in schema (#5564) | https://github.com/apache/seatunnel/commit/eac76b4e50 | 2.3.4 |
[Improve]Remove scheduler in Dynamodb sink (#5248) | https://github.com/apache/seatunnel/commit/9e033a824e | 2.3.4 |
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260) | https://github.com/apache/seatunnel/commit/51c0d709ba | 2.3.4 |
[Bugfix][AmazonDynamoDB] Fix the problem that all table data cannot be obtained (#5146) | https://github.com/apache/seatunnel/commit/09995159a0 | 2.3.3 |
Merge branch 'dev' into merge/cdc | https://github.com/apache/seatunnel/commit/4324ee1912 | 2.3.1 |
[Improve][Project] Code format with spotless plugin. | https://github.com/apache/seatunnel/commit/423b583038 | 2.3.1 |
[improve][api] Refactoring schema parse (#4157) | https://github.com/apache/seatunnel/commit/b2f573a13e | 2.3.1 |
[Improve][build] Give the maven module a human readable name (#4114) | https://github.com/apache/seatunnel/commit/d7cd601051 | 2.3.1 |
[Improve][Project] Code format with spotless plugin. (#4101) | https://github.com/apache/seatunnel/commit/a2ab166561 | 2.3.1 |
[Feature][Connector] add get source method to all source connector (#3846) | https://github.com/apache/seatunnel/commit/417178fb84 | 2.3.1 |
[Feature][API & Connector & Doc] add parallelism and column projection interface (#3829) | https://github.com/apache/seatunnel/commit/b9164b8ba1 | 2.3.1 |
[Hotfix][API] env required option can not set default value (#3584) | https://github.com/apache/seatunnel/commit/c5a23024f6 | 2.3.0 |
[Feature][Connector-V2][AmazonDynamoDB] Add Factory for AmazonDynamoDB (#3348) | https://github.com/apache/seatunnel/commit/a0068efdbf | 2.3.0 |
[Improve][Connector-V2][AmazonDynamoDB] Unified exception for AmazonDynamoDB source & sink connector (#3333) | https://github.com/apache/seatunnel/commit/17bc5adcef | 2.3.0 |
[Connector-V2][Chore] Canonical name for AmazonDynamodb (#3321) | https://github.com/apache/seatunnel/commit/e216eb9a6b | 2.3.0 |
[Feature][Connector-V2] [Amazondynamodb Connector]add amazondynamodb source & sink connnector (#3166) | https://github.com/apache/seatunnel/commit/183bac02f0 | 2.3.0 |