Maxcompute
Maxcompute source connector
Description
Used to read data from Maxcompute.
Key features
Options
name | type | required | default value |
---|---|---|---|
accessId | string | yes | - |
accesskey | string | yes | - |
endpoint | string | yes | - |
project | string | yes | - |
table_name | string | yes | - |
partition_spec | string | no | - |
split_row | int | no | 10000 |
read_columns | Array | no | - |
table_list | Array | No | - |
common-options | string | no | |
schema | config | no |
accessId [string]
accessId
Your Maxcompute accessId which cloud be access from Alibaba Cloud.
accesskey [string]
accesskey
Your Maxcompute accessKey which cloud be access from Alibaba Cloud.
endpoint [string]
endpoint
Your Maxcompute endpoint start with http.
project [string]
project
Your Maxcompute project which is created in Alibaba Cloud.
table_name [string]
table_name
Target Maxcompute table name eg: fake.
partition_spec [string]
partition_spec
This spec of Maxcompute partition table eg:ds='20220101'.
split_row [int]
split_row
Number of rows per split, default: 10000.
read_columns [Array]
read_columns
The columns to be read, if not set, all columns will be read. e.g. ["col1", "col2"]
table_list [Array]
The list of tables to be read, you can use this configuration instead of table_name
.
common options
Source plugin common parameters, please refer to Source Common Options for details.
Examples
Read with table
source {
Maxcompute {
accessId="<your access id>"
accesskey="<your access Key>"
endpoint="<http://service.odps.aliyun.com/api>"
project="<your project>"
table_name="<your table name>"
#partition_spec="<your partition spec>"
#split_row = 10000
#read_columns = ["col1", "col2"]
}
}
Read with table list
source {
Maxcompute {
accessId="<your access id>"
accesskey="<your access Key>"
endpoint="<http://service.odps.aliyun.com/api>"
project="<your project>" # default project
table_list = [
{
table_name = "test_table"
#partition_spec="<your partition spec>"
#split_row = 10000
#read_columns = ["col1", "col2"]
},
{
project = "test_project"
table_name = "test_table2"
#partition_spec="<your partition spec>"
#split_row = 10000
#read_columns = ["col1", "col2"]
}
]
}
}
Changelog
Change Log
Change | Commit | Version |
---|---|---|
[Fix][Connector-V2] Fix maxcompute read with partition spec (#8896) | https://github.com/apache/seatunnel/commit/e62bf6c65 | 2.3.10 |
[Fix][Connector-V2] Fix MaxCompute cannot get project and tableName when use schema (#8865) | https://github.com/apache/seatunnel/commit/a24fa8fef | 2.3.10 |
[Improve] restruct connector common options (#8634) | https://github.com/apache/seatunnel/commit/f3499a6ee | 2.3.10 |
[Feature][Connector-V2] Support maxcompute source with multi-table (#8582) | https://github.com/apache/seatunnel/commit/0f7824292 | 2.3.10 |
[Fix][Connector-V2] Fixed adding table comments (#8514) | https://github.com/apache/seatunnel/commit/edca75b0d | 2.3.10 |
[Improve][Connector-V2] MaxComputeSink support create partition in savemode (#8474) | https://github.com/apache/seatunnel/commit/0b8f9de46 | 2.3.10 |
[Improve][Transform] Rename sql transform table name from 'fake' to 'dual' (#8298) | https://github.com/apache/seatunnel/commit/e6169684f | 2.3.9 |
[Feature][Connector-V2] Support MaxCompute save mode (#8277) | https://github.com/apache/seatunnel/commit/44ea675f1 | 2.3.9 |
[Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef80001 | 2.3.9 |
[Feature][Core] Rename result_table_name /source_table_name to plugin_input/plugin_output (#8072) | https://github.com/apache/seatunnel/commit/c7bbd322d | 2.3.9 |
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03 | 2.3.9 |
[Fix] Fix dead link on seatunnel connectors list url (#7453) | https://github.com/apache/seatunnel/commit/62b4f16f4 | 2.3.8 |
[BugFix][Connector-V2][Maxcompute]fix:Maxcompute sink can't map field(#7164) (#7168) | https://github.com/apache/seatunnel/commit/d5abf8f50 | 2.3.6 |
[Feature] Add unsupported datatype check for all catalog (#5890) | https://github.com/apache/seatunnel/commit/b9791285a | 2.3.4 |
FakeSource support generate different CatalogTable for MultipleTable (#5766) | https://github.com/apache/seatunnel/commit/a8b93805e | 2.3.4 |
[Improve][Common] Introduce new error define rule (#5793) | https://github.com/apache/seatunnel/commit/9d1b2582b | 2.3.4 |
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755) | https://github.com/apache/seatunnel/commit/8de740810 | 2.3.4 |
[Improve][Connector] Add field name to DataTypeConvertor to improve error message (#5782) | https://github.com/apache/seatunnel/commit/ab60790f0 | 2.3.4 |
[Improve][Test] Move MaxCompute test case file (#5786) | https://github.com/apache/seatunnel/commit/38132f515 | 2.3.4 |
[Fix] Fix MaxCompute use not exist SCHEMA option (#5708) | https://github.com/apache/seatunnel/commit/ba4782a67 | 2.3.4 |
[Feature] Support catalog in MaxCompute Source (#5283) | https://github.com/apache/seatunnel/commit/946d89cb9 | 2.3.4 |
[Bugfix][Connector-V2][maxcompute] sink commit with Block not exsits on server (#4725) | https://github.com/apache/seatunnel/commit/2760cae73 | 2.3.2 |
[Bug][Maxcompute] Fix failed to parse some maxcompute type (#3894) | https://github.com/apache/seatunnel/commit/642901f0a | 2.3.1 |
[Improve][build] Give the maven module a human readable name (#4114) | https://github.com/apache/seatunnel/commit/d7cd60105 | 2.3.1 |
[Improve][Project] Code format with spotless plugin. (#4101) | https://github.com/apache/seatunnel/commit/a2ab16656 | 2.3.1 |
[Feature][Connector] add get source method to all source connector (#3846) | https://github.com/apache/seatunnel/commit/417178fb8 | 2.3.1 |
[Feature][API & Connector & Doc] add parallelism and column projection interface (#3829) | https://github.com/apache/seatunnel/commit/b9164b8ba | 2.3.1 |
[Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector (#3640) | https://github.com/apache/seatunnel/commit/80cf8f4e4 | 2.3.0 |