RabbitMQ
RabbitMQ sink connector
Support Those Engines
Spark
Flink
SeaTunnel Zeta
Description
The RabbitMQ sink connector publishes each upstream row as a message to a RabbitMQ
queue, exchange, or routing key. By default messages are sent directly to the named
queue_name via the default exchange; when routing_key (and optionally exchange) is
configured, the connector routes the message through the specified exchange instead.
The sink declares the target queue if it does not already exist, applying the
durable, exclusive, and auto_delete arguments, and reconnects on broker failures
using the standard AMQP recovery options (network_recovery_interval,
topology_recovery_enabled, AUTOMATIC_RECOVERY_ENABLED).
Key features
Options
| name | type | required | default value | description |
|---|---|---|---|---|
| host | string | yes | - | The default host to use for connections. |
| port | int | yes | - | The default port to use for connections. |
| virtual_host | string | yes | - | Virtual host to use when connecting to the broker. |
| username | string | no | - | The AMQP user name to use when connecting to the broker. |
| password | string | no | - | The password to use when connecting to the broker. Must be configured together with username. |
| queue_name | string | yes | - | The queue to write the message to. If routing_key is not configured, the connector publishes to this queue through the default exchange. |
| url | string | no | - | Convenience method for setting host, port, username, password and virtual host in an AMQP URI. |
| routing_key | string | no | - | Routing key used to publish messages. Configure together with exchange to publish through a specific exchange. |
| exchange | string | no | - | Exchange used when routing_key is configured. |
| network_recovery_interval | int | no | - | How long automatic recovery waits before attempting to reconnect, in milliseconds. |
| topology_recovery_enabled | boolean | no | - | Whether to enable topology recovery. |
| AUTOMATIC_RECOVERY_ENABLED | boolean | no | - | Whether to enable connection recovery. The option key is currently uppercase in the connector configuration. |
| connection_timeout | int | no | - | TCP connection establishment timeout, in milliseconds; 0 means infinite. |
| rabbitmq.config | map | no | - | Extra RabbitMQ client parameters. See the official RabbitMQ documentation. |
| common-options | no | - | Sink plugin common parameters. See Sink Common Options. | |
| durable | boolean | no | true | Whether the queue survives a server restart. Used when the connector declares the target queue. |
| exclusive | boolean | no | false | Whether the queue is used only by the current connection and is deleted when the connection closes. |
| auto_delete | boolean | no | false | Whether the queue is deleted automatically when the last consumer unsubscribes. |
| format | string | no | json | The message payload format: json or protobuf. |
| protobuf_schema | string | no | - | Protobuf schema used when format is protobuf. |
| protobuf_message_name | string | no | - | Protobuf message name used when format is protobuf. |
| uri | string | no | - | Legacy alias for url; configure only one. |
| ssl | boolean | no | false | Enable SSL/TLS for host-and-port connections. |
| passive | boolean | no | false | Check that the configured queue exists without declaring it. |
host [string]
The default host to use for connections.
port [int]
The default port to use for connections.
virtual_host [string]
Virtual host – the virtual host to use when connecting to the broker.
username [string]
The AMQP user name to use when connecting to the broker.
password [string]
The password to use when connecting to the broker.
username and password should be configured together.
url [string]
Convenience method for setting the fields in an AMQP URI: host, port, username, password and virtual host.
uri [string]
Legacy alias for url. Configure only one of url and uri.
ssl [boolean]
Enables SSL/TLS for host-and-port configuration. Use url with an amqps:// URI when the URI itself supplies the connection settings.
When url uses an amqps:// URI, the broker certificate is verified against the JVM trust store with hostname verification enabled. Connections that previously relied on the implicit trust-all behavior with self-signed or private-CA certificates must import the broker certificate into the trust store, or they will fail to connect.
queue_name [string]
the queue to write the message to. The value must not be empty or whitespace-only. If routing_key is not configured, the connector publishes messages to this queue through the default exchange.
format [string]
The message payload format. Supported values are json and protobuf. The default value is json.
protobuf_schema [string]
Effective when format is protobuf. Defines the Protobuf schema used to serialize rows into RabbitMQ message payloads.
protobuf_message_name [string]
Effective when format is protobuf. Specifies the Protobuf message name to serialize.
routing_key [string]
The routing key used to publish messages. Configure it together with exchange when you want to publish through a specific exchange instead of directly to queue_name.
exchange [string]
The exchange used when routing_key is configured.
network_recovery_interval [int]
How long automatic recovery waits before attempting to reconnect, in milliseconds.
topology_recovery_enabled [boolean]
If true, enables topology recovery.
AUTOMATIC_RECOVERY_ENABLED [boolean]
If true, enables connection recovery.
The option key is currently uppercase in the connector configuration. Use AUTOMATIC_RECOVERY_ENABLED, not automatic_recovery_enabled.
connection_timeout [int]
TCP connection establishment timeout in milliseconds; 0 means infinite.
rabbitmq.config [map]
In addition to the above parameters that must be specified by the RabbitMQ client, the user can also specify multiple non-mandatory parameters for the client, covering all the parameters specified in the official RabbitMQ document.
Note: the sink currently parses this option but does not apply its entries to the RabbitMQ client. To tune connection behavior, use the dedicated options above such as connection_timeout, network_recovery_interval, AUTOMATIC_RECOVERY_ENABLED, and topology_recovery_enabled.
common options
Sink plugin common parameters. See Sink Common Options for details.
durable
true: The queue will survive on server restart.false: The queue will be deleted on server restart.
exclusive
true: The queue is used only by the current connection and will be deleted when the connection closes.false: The queue can be used by multiple connections.
auto_delete
true: The queue will be deleted automatically when the last consumer unsubscribes.false: The queue will not be automatically deleted.
passive
- false: Declare the queue with the configured durable, exclusive, and auto-delete settings.
- true: Verify that the queue already exists without creating or modifying it. Use this for accounts that can publish but cannot declare queues.
Configuration Notes
- If you configure
username, you must also configurepassword, and vice versa. - Configure only one of
urlanduri.uriis retained for existing configurations; useurlin new configurations. - Set
ssl = truewhen connecting to an AMQPS endpoint withhostandportsettings. host,port,virtual_host, andqueue_nameare required connector options.urlcan additionally provide the AMQP URI used by the RabbitMQ client.durable,exclusive, andauto_deleteare used when the connector declares the target queue.- When
formatisprotobuf, configure bothprotobuf_schemaandprotobuf_message_name.
Example
Write Messages to a Queue
env {
parallelism = 1
job.mode = "STREAMING"
}
source {
FakeSource {
row.num = 10
schema = {
fields {
id = bigint
c_string = string
}
}
}
}
sink {
RabbitMQ {
host = "rabbitmq-e2e"
port = 5672
virtual_host = "/"
username = "guest"
password = "guest"
queue_name = "test1"
}
}
Declare Queue Options
Queue with durable, exclusive, and auto_delete:
env {
parallelism = 1
job.mode = "STREAMING"
}
source {
FakeSource {
row.num = 10
schema = {
fields {
id = bigint
c_string = string
}
}
}
}
sink {
RabbitMQ {
host = "rabbitmq-e2e"
port = 5672
virtual_host = "/"
username = "guest"
password = "guest"
queue_name = "test1"
durable = true
exclusive = false
auto_delete = false
}
}
Write Protobuf Messages to a Queue
sink {
RabbitMQ {
host = "rabbitmq-e2e"
port = 5672
virtual_host = "/"
queue_name = "protobuf_queue"
format = protobuf
protobuf_message_name = Person
protobuf_schema = """
syntax = "proto3";
message Person {
int64 id = 1;
string name = 2;
}
"""
}
}
FAQ
Does RabbitMQ sink support routing to specific exchanges and routing keys?
Yes. The sink publishes messages to RabbitMQ by binding to the target queue or routing configuration specified by queue_name and optional routing parameters.
How does RabbitMQ sink handle network reconnects and timeouts?
Use the dedicated options to tune client connection resilience: AUTOMATIC_RECOVERY_ENABLED enables automatic connection recovery, network_recovery_interval sets how long recovery waits before reconnecting (in ms), topology_recovery_enabled enables topology recovery, and connection_timeout bounds the TCP connection establishment time (in ms). The rabbitmq.config block is accepted for backward compatibility, but its entries are currently not applied by the sink.
Changelog
Change Log
| Change | Commit | Version |
|---|---|---|
| [Fix][connector-rabbitmq] Set default value for durable, exclusive and auto-delete (#9631) | https://github.com/apache/seatunnel/commit/5f9492e62a | 2.3.12 |
| [Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118) | https://github.com/apache/seatunnel/commit/4f5adeb1c7 | 2.3.11 |
| [Improve] rabbit mq options (#8740) | https://github.com/apache/seatunnel/commit/4eec9be012 | 2.3.10 |
| [Improve] restruct connector common options (#8634) | https://github.com/apache/seatunnel/commit/f3499a6eeb | 2.3.10 |
| [Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef800016 | 2.3.9 |
| [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03c | 2.3.9 |
| [Feature][Rabbitmq] Allow configuration of queue durability and deletion policy (#7365) | https://github.com/apache/seatunnel/commit/aabfc8eb78 | 2.3.8 |
| [Hotfix][connector-v2-rabbit] fix rabbit checkpoint exception in Flink mode (#7108) | https://github.com/apache/seatunnel/commit/423a7b142b | 2.3.6 |
| [Feature][Kafka] Support multi-table source read (#5992) | https://github.com/apache/seatunnel/commit/60104602d1 | 2.3.6 |
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755) | https://github.com/apache/seatunnel/commit/8de7408100 | 2.3.4 |
| Support config column/primaryKey/constraintKey in schema (#5564) | https://github.com/apache/seatunnel/commit/eac76b4e50 | 2.3.4 |
| [Bugfix][connector-v2][rabbitmq] Fix reduplicate ack msg bug and code style (#4842) | https://github.com/apache/seatunnel/commit/985fb6642a | 2.3.2 |
| [Hotfix][E2E] Fix RabbitmqIT (#4593) | https://github.com/apache/seatunnel/commit/9bd5403d71 | 2.3.2 |
| Merge branch 'dev' into merge/cdc | https://github.com/apache/seatunnel/commit/4324ee1912 | 2.3.1 |
| [Improve][Project] Code format with spotless plugin. | https://github.com/apache/seatunnel/commit/423b583038 | 2.3.1 |
| [improve][api] Refactoring schema parse (#4157) | https://github.com/apache/seatunnel/commit/b2f573a13e | 2.3.1 |
| [Improve][build] Give the maven module a human readable name (#4114) | https://github.com/apache/seatunnel/commit/d7cd601051 | 2.3.1 |
| [Improve][Project] Code format with spotless plugin. (#4101) | https://github.com/apache/seatunnel/commit/a2ab166561 | 2.3.1 |
| [Feature][Connector] add get source method to all source connector (#3846) | https://github.com/apache/seatunnel/commit/417178fb84 | 2.3.1 |
| [Improve][Connector-V2] Change Connector Custom Config Prefix To Map (#3719) | https://github.com/apache/seatunnel/commit/ef1b8b1bb5 | 2.3.1 |
| [Feature][API & Connector & Doc] add parallelism and column projection interface (#3829) | https://github.com/apache/seatunnel/commit/b9164b8ba1 | 2.3.1 |
| [Hotfix][OptionRule] Fix option rule about all connectors (#3592) | https://github.com/apache/seatunnel/commit/226dc6a119 | 2.3.0 |
| [Feature][Connector-V2][RabbitMQ] Add RabbitMQ source & sink connector (#3312) | https://github.com/apache/seatunnel/commit/4b12691a8d | 2.3.0 |