Maxcompute
Maxcompute 源连接器
引擎支持
Spark
Flink
SeaTunnel Zeta
描述
用于从 Maxcompute 读取数据。连接器支持 AccessKey(accessId/accesskey)认证、STS Token 临时认证,以及阿里云默认凭据链(环境变量、ECS RAM 角色等)免密认证;支持通过 table_name 读取单表、通过 table_list 读取多表,并支持分区表、自定义列和并行读取所需的 split_row。
主要特性
选项
| 名称 | 类型 | 必需 | 默认值 | 说明 |
|---|---|---|---|---|
| accessId | string | 否 | - | 访问 MaxCompute 的 AccessKey ID。 |
| accesskey | string | 否 | - | 访问 MaxCompute 的 AccessKey Secret。 |
| sts_token | string | 否 | - | MaxCompute 临时认证 STS Token。配置 sts_token 时,accessId 与 accesskey 必填。 |
| endpoint | string | 是 | - | MaxCompute 端点,以 http 开头。 |
| project | string | 是 | - | 在阿里云中创建的 MaxCompute 项目。 |
| table_name | string | 未配置 table_list 时必填 | - | 目标 MaxCompute 表名,例如 fake。 |
| schema_name | string | 否 | - | MaxCompute Schema 名称(Project 与 Table 之间的命名空间)。仅当表位于非默认 Schema 时需要设置。 |
| partition_spec | string | 否 | - | MaxCompute 分区表的规范,例如 ds='20220101'。 |
| split_row | int | 否 | 10000 | 每个 split 包含的行数。 |
| read_columns | Array | 否 | - | 要读取的列;不设置时读取全部列,例如 ["col1", "col2"]。 |
| table_list | Array | 否 | - | 要读取的表列表;可替代 table_name 一次读取多张 MaxCompute 表。 |
| tunnel_endpoint | string | 否 | - | MaxCompute Tunnel 服务的自定义端点;未配置时根据区域自动推断。 |
| tunnel_name | string | 否 | - | Tunnel Quota 名称;需同时将 endpoint 与 tunnel_endpoint 配置为 VPC 端点。 |
| connect_timeout_ms | long | 否 | 10000 | ODPS REST 客户端(元数据/catalog 调用)的 HTTP 连接超时,单位毫秒。默认 10000(10 秒)。 |
| read_timeout_ms | long | 否 | 120000 | ODPS REST 客户端(元数据/catalog 调用)的 HTTP 读超时,单位毫秒。默认 120000(120 秒)。 |
| retry_times | int | 否 | 4 | ODPS REST 客户端最大重试次数。默认 4。 |
| tunnel_connect_timeout_ms | long | 否 | 180000 | Tunnel 客户端(批量数据下载)的 HTTP 连接超时,单位毫秒。默认 180000(180 秒)。 |
| tunnel_read_timeout_ms | long | 否 | 300000 | Tunnel 客户端(批量数据下载)的 HTTP 读超时,单位毫秒。默认 300000(300 秒)。 |
| tunnel_retry_times | int | 否 | 4 | Tunnel 客户端最大重试次数。默认 4。 |
| schema | config | 否 | - | 源表结构;未配置 read_columns 时按 schema 中字段读取。 |
| common-options | 否 | - | Source 插件通用参数,例如 plugin_output。 |
accessId [string]
accessId 您的 Maxcompute 密钥 Id.
accesskey [string]
accesskey 您的 Maxcompute 密钥.
sts_token [string]
sts_token 您的 MaxCompute STS Token,用于临时认证。 注意: 如果提供了 sts_token,则必须同时提供 accessId 和 accesskey。
免密认证 (ECS RAM Role, 环境变量等) 要使用免密认证,只需将
accessId、accesskey和sts_token全部留空不填。连接器将自动回退到阿里云默认凭据链 (DefaultCredentialsProvider) 读取凭证(包括环境变量、系统属性、CLI 配置文件、OIDC 以及 ECS RAM 角色)。
endpoint [string]
您的 MaxCompute 端点,以 http 开头。此选项必填,不能是空字符串或仅包含空白字符。空白值会在配置校验阶段被拒绝,不会进入 MaxCompute 连接阶段。
project [string]
project 您在阿里云中创建的 Maxcompute 项目。
table_name [string]
table_name 目标 Maxcompute 表名,例如:fake。
table_name 和 table_list 不能同时配置。读取单表时使用 table_name,读取多表时使用 table_list。
partition_spec [string]
partition_spec Maxcompute 分区表的规范,例如: ds='20220101'。
schema_name [string]
schema_name MaxCompute Schema 名称(Project 与 Table 之间的命名空间)。
仅当表位于 MaxCompute 项目的非默认 Schema 时才需要设置。
参见 Schema 相关操作。
使用 table_list 时,每个条目可以单独指定 schema_name,会覆盖顶层的值。
默认值:不设置(使用项目默认 Schema)。
split_row [int]
split_row 每次拆分的行数,默认值: 10000.
read_columns [Array]
read_columns 要读取的列,如果未设置,则将读取所有列。例如. ["col1", "col2"]
table_list [Array]
要读取的表列表,您可以使用此配置代替 table_name。
每个表配置项都必须包含 table_name,也可以单独覆盖 project、schema_name、partition_spec、split_row 和 read_columns。如果表配置项没有设置这些值,连接器会使用顶层配置。
当一个任务需要用同一组账号、endpoint 和默认 project 读取多张 MaxCompute 表时,可以使用该模式。
tunnel_endpoint [String]
MaxCompute Tunnel 服务的自定义端点。未配置时,连接器会根据区域自动推断默认 Tunnel 端点。
一般只有自定义网络、调试或本地开发时才需要配置,例如 http://maxcompute:8080。
通常,您不需要设置 tunnel_endpoint。仅在自定义网络、调试或本地开发时才需要。
示例值:
https://dt.cn-hangzhou.maxcompute.aliyun.comhttps://dt.ap-southeast-1.maxcompute.aliyun.comhttp://maxcompute:8080
默认值:未设置(从区域自动推断)
tunnel_name [String]
tunnel_name 指定 Tunnel Quota 名称,用于独占资源组。
Tunnel Quota 允许您使用专用的计算资源进行 MaxCompute Tunnel 数据传输,从而提供更好的性能和资源隔离。
重要提示:Tunnel Quota 仅在 VPC(虚拟私有云)端点下生效,暂不支持公共网络访问。使用 tunnel_name 时,必须同时配置 endpoint 和 tunnel_endpoint 为 VPC 端点。
如果未指定,将使用默认的 Tunnel quota。
示例值:
your_tunnel_quota_name
默认值:未设置(使用默认 quota)
客户端超时与重试 MaxCompute 有两个 HTTP 客户端。ODPS REST 客户端负责控制面(表/schema 查询、catalog 列表), 用
connect_timeout_ms、read_timeout_ms、retry_times调整;Tunnel 客户端负责数据面(批量行下载), 用tunnel_*系列调整。单独设置 REST 参数不会改变 Tunnel 客户端的超时。 毫秒值会被转换为整秒,最小值为1000。
connect_timeout_ms [long]
connect_timeout_ms MaxCompute ODPS REST 客户端的 HTTP 连接超时,该客户端处理元数据与 catalog 调用(表/schema 查询、表列表)。单位毫秒。默认 10000(10 秒)。
read_timeout_ms [long]
read_timeout_ms ODPS REST 客户端(元数据/catalog 调用)的 HTTP 读超时,单位毫秒。默认 120000(120 秒)。当项目下表非常多或 schema 极宽导致拉取超时时调大。
retry_times [int]
retry_times ODPS REST 客户端在瞬态失败时的最大重试次数。默认 4。
tunnel_connect_timeout_ms [long]
tunnel_connect_timeout_ms Tunnel 客户端(执行批量数据下载)的 HTTP 连接超时,单位毫秒。默认 180000(180 秒)。
tunnel_read_timeout_ms [long]
tunnel_read_timeout_ms Tunnel 客户端(批量数据下载)的 HTTP 读超时,单位毫秒。默认 300000(300 秒)。当单次读请求超过 5 分钟(大表或大分区)时调大。
tunnel_retry_times [int]
tunnel_retry_times Tunnel 客户端在瞬态失败时的最大重试次数。默认 4。
common options
源插件常用参数, 详见 源通用选项 .
示例
表读取
source {
Maxcompute {
accessId="<your access id>"
accesskey="<your access Key>"
endpoint="<http://service.odps.aliyun.com/api>"
project="<your project>"
table_name="<your table name>"
#tunnel_endpoint="<your tunnel endpoint>"
#partition_spec="<your partition spec>"
#split_row = 10000
#read_columns = ["col1", "col2"]
}
}
使用表列表读取
source {
Maxcompute {
accessId="<your access id>"
accesskey="<your access Key>"
endpoint="<http://service.odps.aliyun.com/api>"
project="<your project>" # default project
#tunnel_endpoint="<your tunnel endpoint>"
table_list = [
{
table_name = "test_table"
#partition_spec="<your partition spec>"
#split_row = 10000
#read_columns = ["col1", "col2"]
},
{
project = "test_project"
table_name = "test_table2"
#partition_spec="<your partition spec>"
#split_row = 10000
#read_columns = ["col1", "col2"]
}
]
}
}
变更日志
Change Log
| Change | Commit | Version |
|---|---|---|
| [Improve][Connector-V2] Validate nonblank MaxCompute source endpoint (#12690) | https://github.com/apache/seatunnel/commit/b3898b62d7 | 3.1.0 |
| [Feature][Connector-V2] Expose MaxCompute client timeout and retry options (#12390) | https://github.com/apache/seatunnel/commit/f80794289d | 3.1.0 |
| [Feature][shade]Refactor the seatunnel-shade module. (#9993) | https://github.com/apache/seatunnel/commit/4ba289595e | 3.0.0 |
| [Feature][Connector-V2] Add tunnel_name option to MaxCompute Connector (#10690) | https://github.com/apache/seatunnel/commit/8e15bd9615 | 3.0.0 |
| [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment (#11131) | https://github.com/apache/seatunnel/commit/63a7b529c5 | 3.0.0 |
| [Docs]Repair document links (#11033) | https://github.com/apache/seatunnel/commit/f3461c9ba9 | 3.0.0 |
| [Feature][Connector-V2][MaxCompute] Support Schema Name and Passwordless Auth (#10831) | https://github.com/apache/seatunnel/commit/78df51cf0f | 3.0.0 |
| [BUG][Connector-V2][MaxCompute] Fix Maxcompute source split stategy (#10845) | https://github.com/apache/seatunnel/commit/17eb09126e | 3.0.0 |
| [Fix][Connector-V2] Use upload session for insert (#10222) | https://github.com/apache/seatunnel/commit/8625828d4a | 3.0.0 |
| [Improve][seatunnel-api] Unify non-relational data source multi-table mode schema configuration parameters (#10370) | https://github.com/apache/seatunnel/commit/3f1fd541fe | 2.3.13 |
| [Fix] Fix CVE-2025-48924 for commons-lang3 (#9935) | https://github.com/apache/seatunnel/commit/c079a967e7 | 2.3.13 |
| [Fix][Core] Add shade module for apache commons lang3 (#9895) | https://github.com/apache/seatunnel/commit/abb9124b05 | 2.3.13 |
| [Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671) | https://github.com/apache/seatunnel/commit/9212a77140 | 2.3.12 |
| [Bug][Connector-V2] NoSuchMethodError caused by Netty version conflict on Spark 3.3.0 (#9632) | https://github.com/apache/seatunnel/commit/4d2b55ce3c | 2.3.12 |
| [Improve][Connector-V2] Replace deprecated createDownloadSession by buildDownloadSession (#9555) | https://github.com/apache/seatunnel/commit/6862945eef | 2.3.12 |
| [Improve][Connector-V2] Add tunnel_endpoint option to MaxCompute source for emulator test (#9548) | https://github.com/apache/seatunnel/commit/b3f3c527ca | 2.3.12 |
| [Improve][Connector-V2] Support maxcompute sink writer upsert/delete action with upsert session mode (#9462) | https://github.com/apache/seatunnel/commit/eb9c8704b9 | 2.3.12 |
| [Improve][Connector-V2] Support maxcompute sink writer with timestamp field type (#9234) | https://github.com/apache/seatunnel/commit/a513c495e3 | 2.3.12 |
| [Feature][Transform] Support define sink column type (#9114) | https://github.com/apache/seatunnel/commit/ab7119e507 | 2.3.11 |
| [Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118) | https://github.com/apache/seatunnel/commit/4f5adeb1c7 | 2.3.11 |
| [Improve] maxcompute options (#9163) | https://github.com/apache/seatunnel/commit/fdacbae1af | 2.3.11 |
| [Fix][Connector-V2] Fix maxcompute write with multi parallelism (#9089) | https://github.com/apache/seatunnel/commit/9426b7ba2c | 2.3.11 |
| [Fix][Connector-V2] Fix maxcompute sink write date less than actual date (#8999) | https://github.com/apache/seatunnel/commit/fc942a599b | 2.3.11 |
| [Fix][Connector-V2] Fix maxcompute read with partition spec (#8896) | https://github.com/apache/seatunnel/commit/e62bf6c65c | 2.3.10 |
| [Fix][Connector-V2] Fix MaxCompute cannot get project and tableName when use schema (#8865) | https://github.com/apache/seatunnel/commit/a24fa8fef6 | 2.3.10 |
| [Improve] restruct connector common options (#8634) | https://github.com/apache/seatunnel/commit/f3499a6eeb | 2.3.10 |
| [Feature][Connector-V2] Support maxcompute source with multi-table (#8582) | https://github.com/apache/seatunnel/commit/0f78242923 | 2.3.10 |
| [Fix][Connector-V2] Fixed adding table comments (#8514) | https://github.com/apache/seatunnel/commit/edca75b0d6 | 2.3.10 |
| [Improve][Connector-V2] MaxComputeSink support create partition in savemode (#8474) | https://github.com/apache/seatunnel/commit/0b8f9de465 | 2.3.10 |
| [Improve][Transform] Rename sql transform table name from 'fake' to 'dual' (#8298) | https://github.com/apache/seatunnel/commit/e6169684fb | 2.3.9 |
| [Feature][Connector-V2] Support MaxCompute save mode (#8277) | https://github.com/apache/seatunnel/commit/44ea675f1e | 2.3.9 |
| [Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef800016 | 2.3.9 |
[Feature][Core] Rename result_table_name/source_table_name to plugin_input/plugin_output (#8072) | https://github.com/apache/seatunnel/commit/c7bbd322db | 2.3.9 |
| [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03c | 2.3.9 |
| [Fix] Fix dead link on seatunnel connectors list url (#7453) | https://github.com/apache/seatunnel/commit/62b4f16f4e | 2.3.8 |
| [BugFix][Connector-V2][Maxcompute]fix:Maxcompute sink can't map field(#7164) (#7168) | https://github.com/apache/seatunnel/commit/d5abf8f506 | 2.3.6 |
| [Feature] Add unsupported datatype check for all catalog (#5890) | https://github.com/apache/seatunnel/commit/b9791285a0 | 2.3.4 |
| FakeSource support generate different CatalogTable for MultipleTable (#5766) | https://github.com/apache/seatunnel/commit/a8b93805ea | 2.3.4 |
| [Improve][Common] Introduce new error define rule (#5793) | https://github.com/apache/seatunnel/commit/9d1b2582b2 | 2.3.4 |
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755) | https://github.com/apache/seatunnel/commit/8de7408100 | 2.3.4 |
[Improve][Connector] Add field name to DataTypeConvertor to improve error message (#5782) | https://github.com/apache/seatunnel/commit/ab60790f0d | 2.3.4 |
| [Improve][Test] Move MaxCompute test case file (#5786) | https://github.com/apache/seatunnel/commit/38132f5158 | 2.3.4 |
| [Fix] Fix MaxCompute use not exist SCHEMA option (#5708) | https://github.com/apache/seatunnel/commit/ba4782a67d | 2.3.4 |
| [Feature] Support catalog in MaxCompute Source (#5283) | https://github.com/apache/seatunnel/commit/946d89cb95 | 2.3.4 |
| [Bugfix][Connector-V2][maxcompute] sink commit with Block not exsits on server (#4725) | https://github.com/apache/seatunnel/commit/2760cae73c | 2.3.2 |
| [Bug][Maxcompute] Fix failed to parse some maxcompute type (#3894) | https://github.com/apache/seatunnel/commit/642901f0a2 | 2.3.1 |
| [Improve][build] Give the maven module a human readable name (#4114) | https://github.com/apache/seatunnel/commit/d7cd601051 | 2.3.1 |
| [Improve][Project] Code format with spotless plugin. (#4101) | https://github.com/apache/seatunnel/commit/a2ab166561 | 2.3.1 |
| [Feature][Connector] add get source method to all source connector (#3846) | https://github.com/apache/seatunnel/commit/417178fb84 | 2.3.1 |
| [Feature][API & Connector & Doc] add parallelism and column projection interface (#3829) | https://github.com/apache/seatunnel/commit/b9164b8ba1 | 2.3.1 |
| [Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector (#3640) | https://github.com/apache/seatunnel/commit/80cf8f4e42 | 2.3.0 |