⭐️
Milvus
Milvus sink connector
Description
This Milvus sink connector write data to Milvus or Zilliz Cloud, it has the following features:
- support read and write data by partition
- support write dynamic schema data from Metadata Column
- json data will be converted to json string and sink as json as well
- retry automatically to bypass ratelimit and grpc limit
Key Features
Data Type Mapping
| Milvus Data Type | SeaTunnel Data Type |
|---|---|
| INT8 | TINYINT |
| INT16 | SMALLINT |
| INT32 | INT |
| INT64 | BIGINT |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| BOOL | BOOLEAN |
| JSON | STRING |
| ARRAY | ARRAY |
| VARCHAR | STRING |
| FLOAT_VECTOR | FLOAT_VECTOR |
| BINARY_VECTOR | BINARY_VECTOR |
| FLOAT16_VECTOR | FLOAT16_VECTOR |
| BFLOAT16_VECTOR | BFLOAT16_VECTOR |
| SPARSE_FLOAT_VECTOR | SPARSE_FLOAT_VECTOR |
Sink Options
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| url | String | Yes | - | The URL to connect to Milvus or Zilliz Cloud. |
| token | String | Yes | - | User:password |
| database | String | No | - | Write data to which database, default is source database. |
| schema_save_mode | enum | No | CREATE_SCHEMA_WHEN_NOT_EXIST | Auto create table when table not exist. |
| enable_auto_id | boolean | No | false | Primary key column enable autoId. |
| enable_upsert | boolean | No | false | Upsert data not insert. |
| enable_dynamic_field | boolean | No | true | Enable create table with dynamic field. |
| batch_size | int | No | 1000 | Write batch size. |
| partition_key | String | No | Milvus partition key field |
Task Example
sink {
Milvus {
url = "http://127.0.0.1:19530"
token = "username:password"
batch_size = 1000
}
}
Changelog
Change Log
| Change | Commit | Version |
|---|---|---|
| [Fix][Connector-V2] Fix load state check in MilvusSourceReader to consider partition-level status (#8937) | https://github.com/apache/seatunnel/commit/bde235090 | 2.3.10 |
| [Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef80001 | 2.3.9 |
| [Improve][Core] Refactor common options of column/row (#7911) | https://github.com/apache/seatunnel/commit/d1582afee | 2.3.9 |
| [Feature][connector-milvus] update milvus connector to support dynamic schema, failed retry, etc. (#7885) | https://github.com/apache/seatunnel/commit/6a31f9172 | 2.3.9 |
| [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03 | 2.3.9 |
| [Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700) | https://github.com/apache/seatunnel/commit/e2fb67957 | 2.3.8 |
| [Improve][Connector-V2] Optimize milvus code (#7691) | https://github.com/apache/seatunnel/commit/1eddb8e1b | 2.3.8 |
| [Improve][Connector-V2] Optimize milvus-connector config code (#7658) | https://github.com/apache/seatunnel/commit/f831f7a5e | 2.3.8 |
| [Improve][Connector-V2] update vectorType (#7446) | https://github.com/apache/seatunnel/commit/1bba72385 | 2.3.8 |
| [Improve][API] Move catalog open to SaveModeHandler (#7439) | https://github.com/apache/seatunnel/commit/8c2c5c79a | 2.3.8 |
| [Feature][Connector-V2] Fake Source support produce vector data (#7401) | https://github.com/apache/seatunnel/commit/6937d10ac | 2.3.8 |
| [Feature][Connector-V2][Milvus] Support Milvus source & sink (#7158) | https://github.com/apache/seatunnel/commit/0c69b9166 | 2.3.6 |