跳到主要内容
版本:2.3.10

Milvus

Milvus数据接收器

描述

Milvus sink连接器将数据写入Milvus或Zilliz Cloud,它具有以下功能:

  • 支持按分区读写数据
  • 支持从元数据列写入动态模式数据
  • json数据将转换为json字符串进行写入
  • 自动重试以绕过 ratelimit 限制 和 grpc 限制

主要特性

##数据类型映射

Milvus数据类型SeaTunnel 数据类型
INT8TINYINT
INT16SMALLINT
INT32INT
INT64BIGINT
FLOATFLOAT
DOUBLEDOUBLE
BOOLBOOLEAN
JSONSTRING
ARRAYARRAY
VARCHARSTRING
FLOAT_VECTORFLOAT_VECTOR
BINARY_VECTORBINARY_VECTOR
FLOAT16_VECTORFLOAT16_VECTOR
BFLOAT16_VECTORBFLOAT16_VECTOR
SPARSE_FLOAT_VECTORSPARSE_FLOAT_VECTOR

Sink 选项

名字类型是否必传默认值描述
urlString-连接到Milvus或Zilliz Cloud的URL。
tokenString-用户:密码
databaseString-将数据写入哪个数据库,默认为源数据库。
schema_save_modeenumCREATE_SCHEMA_WHEN_NOT_EXIST当表不存在时自动创建表。
enable_auto_idbooleanfalse主键列启用autoId。
enable_upsertbooleanfalse是否启用upsert。
enable_dynamic_fieldbooleantrue是否启用带动态字段的创建表。
batch_sizeint1000写入批大小。
partition_keyStringMilvus分区键字段

任务示例

sink {
Milvus {
url = "http://127.0.0.1:19530"
token = "username:password"
batch_size = 1000
}
}

变更日志

Change Log
ChangeCommitVersion
[Fix][Connector-V2] Fix load state check in MilvusSourceReader to consider partition-level status (#8937)https://github.com/apache/seatunnel/commit/bde2350902.3.10
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef800012.3.9
[Improve][Core] Refactor common options of column/row (#7911)https://github.com/apache/seatunnel/commit/d1582afee2.3.9
[Feature][connector-milvus] update milvus connector to support dynamic schema, failed retry, etc. (#7885)https://github.com/apache/seatunnel/commit/6a31f91722.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d032.3.9
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700)https://github.com/apache/seatunnel/commit/e2fb679572.3.8
[Improve][Connector-V2] Optimize milvus code (#7691)https://github.com/apache/seatunnel/commit/1eddb8e1b2.3.8
[Improve][Connector-V2] Optimize milvus-connector config code (#7658)https://github.com/apache/seatunnel/commit/f831f7a5e2.3.8
[Improve][Connector-V2] update vectorType (#7446)https://github.com/apache/seatunnel/commit/1bba723852.3.8
[Improve][API] Move catalog open to SaveModeHandler (#7439)https://github.com/apache/seatunnel/commit/8c2c5c79a2.3.8
[Feature][Connector-V2] Fake Source support produce vector data (#7401)https://github.com/apache/seatunnel/commit/6937d10ac2.3.8
[Feature][Connector-V2][Milvus] Support Milvus source & sink (#7158)https://github.com/apache/seatunnel/commit/0c69b91662.3.6