跳到主要内容
版本:3.1.0

RocketMQ

RocketMQ 源连接器

支持的 Apache RocketMQ 版本​

  • 4.9.0 或更新版本

引擎支持​

Spark
Flink
SeaTunnel Zeta

主要特性​

描述​

从 Apache RocketMQ topic 读取消息。连接器既可以用一套 schema 读取一个或多个 topic,也可以通过 tables_configs 读取多张不同结构的表。

源参数​

参数名类型是否必填默认值描述
name.srv.addrString是-RocketMQ NameServer 地址,例如 localhost:9876。
topicsString否-topic 名称,多个 topic 使用逗号分隔,例如 "topic_a,topic_b"。topics、tables_configs 和 table_list 只能配置其中一个。
tables_configsList否-多表读取配置。每一项必须包含 topics,并可配置 format、schema、tags、start.mode、start.mode.timestamp、start.mode.offsets 和 ignore_parse_errors。
table_listList否-已废弃,请使用 tables_configs。
tagsString否-tag 名称,多个 tag 使用逗号分隔。只消费 RocketMQ tag 与配置值完全匹配的消息。
acl.enabledBoolean否false是否启用 RocketMQ ACL 鉴权。
access.keyString否-访问密钥。acl.enabled = true 时必填。
secret.keyString否-秘密密钥。acl.enabled = true 时必填。
batch.sizeint否100每次最多拉取的消息数。
consumer.groupString否SeaTunnel-Consumer-GroupRocketMQ 消费者组 ID。
commit.on.checkpointBoolean否true是否在 SeaTunnel checkpoint 完成后提交消费位点。
schemaconfig否-消息结构。详情请参考 Schema 特性。不配置时,连接器按文本读取消息体。
formatString否json消息格式。支持 json 和 text。
field.delimiterString否,format = text 时使用的字段分隔符。
start.modeString否CONSUME_FROM_GROUP_OFFSETS启动消费位置。支持:CONSUME_FROM_LAST_OFFSET、CONSUME_FROM_FIRST_OFFSET、CONSUME_FROM_GROUP_OFFSETS、CONSUME_FROM_TIMESTAMP、CONSUME_FROM_SPECIFIC_OFFSETS。
start.mode.offsetsMap否-start.mode = CONSUME_FROM_SPECIFIC_OFFSETS 时必填。key 格式为 topic-queueId,例如 test_topic-0。
start.mode.timestampLong否-start.mode = CONSUME_FROM_TIMESTAMP 时必填,单位是毫秒时间戳。
partition.discovery.interval.millislong否-1动态发现 topic 和分区的间隔,单位毫秒。-1 表示不启用动态发现。
ignore_parse_errorsBoolean否false是否跳过解析失败的 JSON 消息。
consumer.poll.timeout.millislong否5000拉取消息的超时时间,单位毫秒。
common-optionsconfig否-源连接器通用参数,详情请参考 源通用参数。

参数说明​

启动消费位置​

start.mode 用来控制从哪里开始读:

  • CONSUME_FROM_GROUP_OFFSETS:从消费者组已提交的位点开始读。
  • CONSUME_FROM_FIRST_OFFSET:从最早可用位点开始读。
  • CONSUME_FROM_LAST_OFFSET:从最新位点开始读。
  • CONSUME_FROM_TIMESTAMP:从 start.mode.timestamp 对应时间之后的第一条消息开始读。
  • CONSUME_FROM_SPECIFIC_OFFSETS:从 start.mode.offsets 指定的位点开始读。

当 start.mode = CONSUME_FROM_TIMESTAMP 时,start.mode.timestamp 必须是非负的毫秒时间戳,并且不能晚于任务运行时的当前时间。

start.mode = "CONSUME_FROM_SPECIFIC_OFFSETS"
start.mode.offsets = {
test_topic-0 = 50
}
start.mode = "CONSUME_FROM_TIMESTAMP"
start.mode.timestamp = 1667179890315

消息格式​

当 format = json 时,请配置 schema,SeaTunnel 会按 schema 将 JSON 消息体解析成有类型的字段。配置 ignore_parse_errors = true 后,遇到无法解析的 JSON 消息会跳过,而不是让任务失败。

当 format = text 时,SeaTunnel 会按 field.delimiter 拆分消息体,并按照 schema 字段顺序映射数据。如果不配置 schema,消息体会作为单个文本值读取。

标签​

tags 使用逗号分隔,例如 tag_a,tag_b。连接器会把拉取到的消息 tag 和这些值逐个做精确匹配,因此这里不要使用 RocketMQ 的 tag 表达式语法,例如 tag_a || tag_b。在多表读取任务中,每个 tables_configs 条目都可以配置自己的 tags 过滤条件。

多表读取​

当不同 topic 的字段结构不一样时,使用 tables_configs。每一项都必须包含 topics,并且可以单独配置 schema、format、tags 和启动消费位置。如果没有配置 schema.table,输出表名默认使用 topic 名称。

topics、tables_configs 和已废弃的 table_list 互斥,只能配置其中一个。在 tables_configs 中,单个条目未配置的参数会沿用顶层默认值,因此每个条目只需要覆盖该 topic 特有的 schema、tag 或启动位置。

如果某个 tables_configs 条目使用 start.mode = CONSUME_FROM_TIMESTAMP,必须同时配置 start.mode.timestamp。如果使用 start.mode = CONSUME_FROM_SPECIFIC_OFFSETS,必须同时配置非空的 start.mode.offsets。

任务示例​

读取 JSON 消息​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
topics = "test_topic_json"
plugin_output = "rocketmq_table"
format = json
schema = {
fields {
id = bigint
c_string = string
c_int = int
c_timestamp = timestamp
}
}
}
}

sink {
Console {
plugin_input = "rocketmq_table"
}
}

按 tag 读取文本消息​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
topics = "test_topic_text"
plugin_output = "rocketmq_table"
format = text
field.delimiter = ","
tags = "tag_a,tag_b"
schema = {
fields {
id = bigint
content = string
}
}
}
}

sink {
Console {
plugin_input = "rocketmq_table"
}
}

从指定 offset 读取​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
topics = "test_topic_source"
plugin_output = "rocketmq_table"
format = json
start.mode = "CONSUME_FROM_SPECIFIC_OFFSETS"
start.mode.offsets = {
test_topic_source-0 = 50
}
schema = {
fields {
id = bigint
}
}
}
}

sink {
Console {
plugin_input = "rocketmq_table"
}
}

读取多个不同结构的 topic​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
start.mode = "CONSUME_FROM_LAST_OFFSET"
tables_configs = [
{
topics = "test_topic_multi_a"
start.mode = "CONSUME_FROM_FIRST_OFFSET"
format = json
schema = {
fields {
id = bigint
c_string = string
}
}
},
{
topics = "test_topic_multi_b"
start.mode = "CONSUME_FROM_FIRST_OFFSET"
tags = "tag_b"
format = json
schema = {
table = "rocketmq_multi_custom"
fields {
id = bigint
description = string
}
}
}
]
}
}

sink {
Console {}
}

从指定时间戳读取​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
topics = "test_topic_source"
plugin_output = "rocketmq_table"
format = json
start.mode = "CONSUME_FROM_TIMESTAMP"
start.mode.timestamp = 1667179890315
schema = {
fields {
id = bigint
}
}
}
}

sink {
Console {
plugin_input = "rocketmq_table"
}
}

FAQ​

RocketMQ 源连接器如何支持多 Topic 不同 Schema 的读取?​

可以通过配置 tables_configs 数组来实现。每个配置项可独立定义该 Topic 的 topics、schema、format 以及 tags,SeaTunnel 将并发消费并独立解析为对应的表数据流。

RocketMQ 源连接器支持哪些起始消费位点模式?​

通过 start.mode 参数支持从最早位点(CONSUME_FROM_FIRST_OFFSET)、最新位点(CONSUME_FROM_LAST_OFFSET)或指定时间戳位点(CONSUME_FROM_TIMESTAMP)开始读取数据。

作业发生故障恢复时如何保证消费位点一致性?​

SeaTunnel 会周期性对消费者队列位点进行 Checkpoint 状态持久化。当任务发生异常或节点故障重启时,源连接器将严格从上一次成功提交的 Checkpoint 位点重新恢复消费。

变更日志​

Change Log
ChangeCommitVersion
[Fix][Connector-V2][RocketMQ] Close consumer client before interrupting its executor (#12099)https://github.com/apache/seatunnel/commit/934276b2f73.1.0
[Fix][Connector-V2] Keep RocketMQ group offsets when a later topic has no retry route (#12648)https://github.com/apache/seatunnel/commit/07f32fffbd3.1.0
[Improve][Connector-V2] Correct two RocketMQ comments that no longer hold (#12438)https://github.com/apache/seatunnel/commit/524312d7413.1.0
[Improve][Connector-V2] Note the multi-topic early return in currentOffsets (#12417)https://github.com/apache/seatunnel/commit/6934fd00623.1.0
[Fix][Connector-V2] Surface RocketMQ group-offset lookup failures instead of rewinding (#12349)https://github.com/apache/seatunnel/commit/b70652be8a3.1.0
[Fix][Zeta] Reuse fixed slots after master failover (#11458)https://github.com/apache/seatunnel/commit/a9cda80f4b3.0.0
[Improve][Common] Add HashUtils.bucketIndex for hash-to-bucket routing (#11987)https://github.com/apache/seatunnel/commit/be53a1d3d13.0.0
[Improve][Connector-V2] Migrate RocketMQ Source validation to declarative OptionRule (#11158)https://github.com/apache/seatunnel/commit/c8fb49358a3.0.0
Test/rocketmq restore e2e (#10778)https://github.com/apache/seatunnel/commit/6d2a1046593.0.0
[Improve][Connector-V2] Complete OptionRule declarations for RocketMQ source and sink (#10701)https://github.com/apache/seatunnel/commit/7becf69b703.0.0
[Feature][Connector-V2] Support multi-table read for RocketMQ source (#10619)https://github.com/apache/seatunnel/commit/85615468b73.0.0
[Fix][Core] Add shade module for apache commons lang3 (#9895)https://github.com/apache/seatunnel/commit/abb9124b052.3.13
[Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671)https://github.com/apache/seatunnel/commit/9212a771402.3.12
[improve] rocketmq options (#9251)https://github.com/apache/seatunnel/commit/4cbe3b91722.3.12
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Improve][Connector-V2] RocketMQ Source add message tag config (#8825)https://github.com/apache/seatunnel/commit/5913e8c35f2.3.10
[Improve][Connector-V2] Add optional flag for rocketmq connector to skip parse errors instead of failing (#8737)https://github.com/apache/seatunnel/commit/701f17b5d42.3.10
[Improve][Connector-V2] RocketMQ Sink add message tag config (#7996)https://github.com/apache/seatunnel/commit/97a1b00e482.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Fix][Connector-V2] Fix some throwable error not be caught (#7657)https://github.com/apache/seatunnel/commit/e19d73282e2.3.8
[Feature][Kafka] Support multi-table source read (#5992)https://github.com/apache/seatunnel/commit/60104602d12.3.6
[Fix][connector-rocketmq] commit a correct offset to broker & reduce ThreadInterruptedException log (#6668)https://github.com/apache/seatunnel/commit/b7480e1a892.3.6
[fix][connector-rocketmq]Fix a NPE problem when checkpoint.interval is set too small(#6624) (#6625)https://github.com/apache/seatunnel/commit/6e0c81d4922.3.5
[Test][E2E] Add thread leak check for connector (#5773)https://github.com/apache/seatunnel/commit/1f2f3fc5f02.3.4
[Fix][Connector] Rocketmq source startOffset greater than endOffset error (#6287)https://github.com/apache/seatunnel/commit/cd44b5894e2.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)https://github.com/apache/seatunnel/commit/51c0d709ba2.3.4
[Improve][pom] Formatting pom (#4761)https://github.com/apache/seatunnel/commit/1d6d3815ec2.3.2
[Hotfix][Connector-V2][RocketMQ] Fix rocketmq spark e2e test cases (#4583)https://github.com/apache/seatunnel/commit/e711f6ef4c2.3.2
[Feature][Connector-V2] Add rocketmq source and sink (#4007)https://github.com/apache/seatunnel/commit/e3338975522.3.2