Skip to main content
Version: Next

Milvus

Milvus source connector

Description​

This Milvus source connector reads data from Milvus or Zilliz Cloud, it has the following features:

  • support read and write data by partition
  • support read dynamic schema data into Metadata Column
  • json data will be converted to json string and sink as json as well
  • retry automatically to bypass ratelimit and grpc limit

Key Features​

Data Type Mapping​

Milvus Data TypeSeaTunnel Data Type
INT8TINYINT
INT16SMALLINT
INT32INT
INT64BIGINT
FLOATFLOAT
DOUBLEDOUBLE
BOOLBOOLEAN
JSONSTRING
ARRAYARRAY
VARCHARSTRING
FLOAT_VECTORFLOAT_VECTOR
BINARY_VECTORBINARY_VECTOR
FLOAT16_VECTORFLOAT16_VECTOR
BFLOAT16_VECTORBFLOAT16_VECTOR
SPARSE_FLOAT_VECTORSPARSE_FLOAT_VECTOR

Source Options​

NameTypeRequiredDefaultDescription
urlStringYes-The URL to connect to Milvus or Zilliz Cloud.
tokenStringYes-User:password
databaseStringYesdefaultRead data from which database.
collectionStringNo-If set, will only read one collection, otherwise will read all collections under database.

Task Example​

source {
  Milvus {
    url = "http://127.0.0.1:19530"
    token = "username:password"
    database = "default"
  }
}

Changelog​

Change Log
ChangeCommitVersion
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Improve][Core] Refactor common options of column/row (#7911)https://github.com/apache/seatunnel/commit/d1582afee62.3.9
[Feature][connector-milvus] update milvus connector to support dynamic schema, failed retry, etc. (#7885)https://github.com/apache/seatunnel/commit/6a31f917292.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700)https://github.com/apache/seatunnel/commit/e2fb6795772.3.8
[Improve][Connector-V2] Optimize milvus code (#7691)https://github.com/apache/seatunnel/commit/1eddb8e1b12.3.8
[Improve][Connector-V2] Optimize milvus-connector config code (#7658)https://github.com/apache/seatunnel/commit/f831f7a5ec2.3.8
[Improve][Connector-V2] update vectorType (#7446)https://github.com/apache/seatunnel/commit/1bba72385b2.3.8
[Improve][API] Move catalog open to SaveModeHandler (#7439)https://github.com/apache/seatunnel/commit/8c2c5c79a12.3.8
[Feature][Connector-V2] Fake Source support produce vector data (#7401)https://github.com/apache/seatunnel/commit/6937d10ac32.3.8
[Feature][Connector-V2][Milvus] Support Milvus source & sink (#7158)https://github.com/apache/seatunnel/commit/0c69b9166e2.3.6