跳到主要内容
版本：2.3.12

Redis

Redis sink connector

描述​

用于将数据写入 Redis。

主要功能​

选项​

nametyperequireddefault value
hoststringmode=single时必须-
portintno6379
keystringyes-
data_typestringyes-
batch_sizeintno10
userstringno-
authstringno-
db_numintno0
modestringnosingle
nodeslistyes when mode=cluster-
formatstringnojson
expirelongno-1
support_custom_keybooleannofalse
value_fieldstringno-
hash_key_fieldstringno-
hash_value_fieldstringno-
common-optionsno-

host [string]​

Redis 主机地址

port [int]​

Redis 端口

key [string]​

要写入 Redis 的键值。

例如，如果想使用上游数据中的某个字段值作为键值，可以将该字段名称指定给 key。

上游数据如下：

codedatasuccess
200获取成功true
500内部错误false

如果将字段名称指定为 code 并将 data_type 设置为 key，将有两个数据写入 Redis：

  1. 200 -> {code: 200, data: 获取成功, success: true}
  2. 500 -> {code: 500, data: 内部错误, success: false}

如果将字段名称指定为 value 并将 data_type 设置为 key，则由于上游数据的字段中没有 value 字段，将只有一个数据写入 Redis：

  1. value -> {code: 500, data: 内部错误, success: false}

请参见 data_type 部分以了解具体的写入规则。

当然，这里写入的数据格式只是以 json 为例，具体格式以用户配置的 format 为准。

data_type [string]​

Redis 数据类型，支持 key hash list set zset

  • key

每个来自上游的数据都会更新到配置的 key，这意味着后面的数据会覆盖前面的数据，只有最后的数据会存储在该 key 中。

  • hash

每个来自上游的数据会根据字段拆分并写入 hash key，后面的数据会覆盖前面的数据。

  • list

每个来自上游的数据都会被添加到配置的 list key 中。

  • set

每个来自上游的数据都会被添加到配置的 set key 中。

  • zset

每个来自上游的数据都会以权重为 1 的方式添加到配置的 zset key 中。因此，zset 中数据的顺序基于数据的消费顺序。

user [string]​

Redis 认证用户，连接加密集群时需要

auth [string]​

Redis 认证密码，连接加密集群时需要

db_num [int]​

Redis 数据库索引 ID，默认连接到 db 0

mode [string]​

Redis 模式，single 或 cluster，默认是 single

nodes [list]​

Redis 节点信息，在集群模式下使用，必须按如下格式：

["host1:port1", "host2:port2"]

format [string]​

上游数据的格式，目前只支持 json，以后会支持 text，默认 json。

当你指定格式为 json 时，例如：

上游数据如下：

codedatasuccess
200获取成功true

连接器会生成如下数据并写入 Redis：

{"code":  200, "data":  "获取成功", "success":  "true"}

expire [long]​

设置 Redis 的过期时间，单位为秒。默认值为 -1，表示键不会自动过期。

support_custom_key [boolean]​

设置为true，表示启用自定义Key。

上游数据如下：

codedatasuccess
200获取成功true
500内部错误false

可以使用{和}符号自定义Redis键名，{}中的字段名会被解析替换为上游数据中的某个字段值，例如：将字段名称指定为 {code} 并将 data_type 设置为 key，将有两个数据写入 Redis：

  1. 200 -> {code: 200, data: 获取成功, success: true}
  2. 500 -> {code: 500, data: 内部错误, success: false}

Redis键名可以由固定部分和变化部分组成，通过Redis分组符号:连接，例如：将字段名称指定为 code:{code} 并将 data_type 设置为 key，将有两个数据写入 Redis：

  1. code:200 -> {code: 200, data: 获取成功, success: true}
  2. code:500 -> {code: 500, data: 内部错误, success: false}

value_field [string]​

要写入Redis的值的字段， data_type 支持 key list set zset.

当你指定Redis键名字段key指定为 value，值字段value_field指定为data，并将data_type指定为key时,

上游数据如下：

codedatasuccess
200获取成功true

如下的数据会被写入Redis:

  1. value -> 获取成功

hash_key_field [string]​

要写入Redis的hash键字段, data_type 支持 hash

hash_value_field [string]​

要写入Redis的hash值字段, data_type 支持 hash

当你指定Redis键名字段key指定为 value，hash键字段hash_key_field指定为data，hash值字段hash_value_field指定为success，并将data_type指定为hash时,

上游数据如下：

codedatasuccess
200获取成功true

如下的数据会被写入Redis:

  1. value -> 获取成功 | true

common options​

Sink 插件通用参数，请参考 Sink Common Options 获取详情

示例​

简单示例：

Redis {
  host = localhost
  port = 6379
  key = age
  data_type = list
}

自定义Key示例：

Redis {
  host = localhost
  port = 6379
  key = "name:{name}"
  support_custom_key = true
  data_type = key
}

自定义Value示例：

Redis {
  host = localhost
  port = 6379
  key = person
  value_field = "name"
  data_type = key
}

自定义HashKey和HashValue示例：

Redis {
  host = localhost
  port = 6379
  key = person
  hash_key_field = "name"
  hash_value_field = "age"
  data_type = hash
}

变更日志​

Change Log
ChangeCommitVersion
[Improve][Connector-V2] Use key_field_name option when reading Redis hash data (#9642)https://github.com/apache/seatunnel/commit/5d214a73052.3.12
[Feature][Redis] Add redis key into the result record (#9574)https://github.com/apache/seatunnel/commit/6e8b7c5da52.3.12
[Fix][Connector-Redis] Redis did not write successfully, but the task did not fail (#9055)https://github.com/apache/seatunnel/commit/07510ed9372.3.11
[hotfix][redis] fix npe cause by null host parameter (#8881)https://github.com/apache/seatunnel/commit/7bd58651652.3.10
[Improve][Redis] Optimized Redis connection params (#8841)https://github.com/apache/seatunnel/commit/e56f06cdf02.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[improve] update Redis connector config option (#8631)https://github.com/apache/seatunnel/commit/f1c313eea62.3.10
[Feature][Redis] Flush data when the time reaches checkpoint.interval and update test case (#8308)https://github.com/apache/seatunnel/commit/e15757bcd72.3.9
Revert "[Feature][Redis] Flush data when the time reaches checkpoint interval" and "[Feature][CDC] Add 'schema-changes.enabled' options" (#8278)https://github.com/apache/seatunnel/commit/fcb29382862.3.9
[Feature][Redis] Flush data when the time reaches checkpoint.interval (#8198)https://github.com/apache/seatunnel/commit/2e24941e6a2.3.9
[Hotfix] Fix redis sink NPE (#8171)https://github.com/apache/seatunnel/commit/6b9074e7692.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Connector-Redis] Redis connector support delete data (#7994)https://github.com/apache/seatunnel/commit/02a35c39792.3.9
[Improve][Connector-V2] Redis support custom key and value (#7888)https://github.com/apache/seatunnel/commit/ef2c3c72832.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[improve][Redis]Redis scan command supports versions 5, 6, 7 (#7666)https://github.com/apache/seatunnel/commit/6e70cbe3342.3.8
[Improve][Connector] Add multi-table sink option check (#7360)https://github.com/apache/seatunnel/commit/2489f6446b2.3.7
[Feature][Core] Support using upstream table placeholders in sink options and auto replacement (#7131)https://github.com/apache/seatunnel/commit/c4ca74122c2.3.6
[Improve][Redis] Redis reader use scan cammnd instead of keys, single mode reader/writer support batch (#7087)https://github.com/apache/seatunnel/commit/be37f05c072.3.6
[Feature][Kafka] Support multi-table source read (#5992)https://github.com/apache/seatunnel/commit/60104602d12.3.6
[Improve][Connector-V2]Support multi-table sink feature for redis (#6314)https://github.com/apache/seatunnel/commit/fed89ae3fc2.3.5
[Feature][Core] Upgrade flink source translation (#5100)https://github.com/apache/seatunnel/commit/5aabb14a942.3.4
[Feature][Connector-V2] Support TableSourceFactory/TableSinkFactory on redis (#5901)https://github.com/apache/seatunnel/commit/e84dcb8c102.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[Improve][Connector-v2][Redis] Redis support select db (#5570)https://github.com/apache/seatunnel/commit/77fbbbd0ee2.3.4
Support config column/primaryKey/constraintKey in schema (#5564)https://github.com/apache/seatunnel/commit/eac76b4e502.3.4
[Feature][Connector-v2][RedisSink]Support redis to set expiration time. (#4975)https://github.com/apache/seatunnel/commit/b5321ff1d22.3.3
Merge branch 'dev' into merge/cdchttps://github.com/apache/seatunnel/commit/4324ee19122.3.1
[Improve][Project] Code format with spotless plugin.https://github.com/apache/seatunnel/commit/423b5830382.3.1
[improve][api] Refactoring schema parse (#4157)https://github.com/apache/seatunnel/commit/b2f573a13e2.3.1
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector] add get source method to all source connector (#3846)https://github.com/apache/seatunnel/commit/417178fb842.3.1
[Hotfix][OptionRule] Fix option rule about all connectors (#3592)https://github.com/apache/seatunnel/commit/226dc6a1192.3.0
[Improve][Connector-V2][Redis] Unified exception for redis source & sink exception (#3517)https://github.com/apache/seatunnel/commit/205f7825852.3.0
options in conditional need add to required or optional options (#3501)https://github.com/apache/seatunnel/commit/51d5bcba102.3.0
[feature][api] add option validation for the ReadonlyConfig (#3417)https://github.com/apache/seatunnel/commit/4f824fea362.3.0
[Feature][Redis Connector V2] Add Redis Connector Option Rules & Improve Redis Connector doc (#3320)https://github.com/apache/seatunnel/commit/1c10aacb302.3.0
[Connector-V2][ElasticSearch] Add ElasticSearch Source/Sink Factory (#3325)https://github.com/apache/seatunnel/commit/38254e3f262.3.0
[Improve][Connector-V2][Redis] Support redis cluster connection & user authentication (#3188)https://github.com/apache/seatunnel/commit/c7275a49cc2.3.0
[DEV][Api] Replace SeaTunnelContext with JobContext and remove singleton pattern (#2706)https://github.com/apache/seatunnel/commit/cbf82f755c2.2.0-beta
[Feature][Connector-V2] Add redis sink connector (#2647)https://github.com/apache/seatunnel/commit/71a9e4b0192.2.0-beta
[#2606]Dependency management split (#2630)https://github.com/apache/seatunnel/commit/fc047be69b2.2.0-beta
[Feature][Connector-V2] Add redis source connector (#2569)https://github.com/apache/seatunnel/commit/405f7d6f992.2.0-beta