Milvus

Milvus source connector

Description

This Milvus source connector reads data from Milvus or Zilliz Cloud, it has the following features:

• support read and write data by partition
• support read dynamic schema data into Metadata Column
• json data will be converted to json string and sink as json as well
• retry automatically to bypass ratelimit and grpc limit

Key Features

• batch
• exactly-once
• column projection

Data Type Mapping

Milvus Data Type	SeaTunnel Data Type
INT8	TINYINT
INT16	SMALLINT
INT32	INT
INT64	BIGINT
FLOAT	FLOAT
DOUBLE	DOUBLE
BOOL	BOOLEAN
JSON	STRING
ARRAY	ARRAY
VARCHAR	STRING
FLOAT_VECTOR	FLOAT_VECTOR
BINARY_VECTOR	BINARY_VECTOR
FLOAT16_VECTOR	FLOAT16_VECTOR
BFLOAT16_VECTOR	BFLOAT16_VECTOR
SPARSE_FLOAT_VECTOR	SPARSE_FLOAT_VECTOR

Source Options

Name	Type	Required	Default	Description
url	String	Yes	-	The URL to connect to Milvus or Zilliz Cloud.
token	String	Yes	-	User:password
database	String	Yes	default	Read data from which database.
collection	String	No	-	If set, will only read one collection, otherwise will read all collections under database.

Task Example

source {
  Milvus {
    url = "http://127.0.0.1:19530"
    token = "username:password"
    database = "default"
  }
}

Changelog