Typesense
Typesense 源连接器
支持的引擎
SeaTunnel Zeta
描述
从 Typesense collection 读取文档。该 source 支持有界批读取,也可以通过 query 传入
Typesense 查询参数。
Source 是有界读取:每次作业只会读取一次匹配 query 的全部文档,然后结束。如果需要变更捕获,请在接收端借助 Typesense 自身的变化追踪机制实现。
主要特性
选项
| 名称 | 类型 | 必填 | 默认值 |
|---|---|---|---|
| hosts | array | 是 | - |
| collection | string | 是 | - |
| schema | config | 是 | - |
| api_key | string | 是 | - |
| protocol | string | 否 | http |
| query | string | 否 | - |
| batch_size | int | 否 | 100 |
| common-options | 否 | - |
hosts [array]
Typesense 的访问地址,格式为 host:port,例如:["typesense-01:8108"]。支持配置多个地址。配置多个节点时,Source 会把搜索请求发往第一个可达节点,列表本身不用于并行扫描分片。
collection [string]
要读取的 Typesense collection 名,例如:companies。
schema [config]
typesense 需要读取的列。有关更多信息,请参阅:guide。
api_key [string]
Typesense 安全认证的 api_key。请把它当作敏感凭据处理;在共享环境运行时,建议通过作业密钥或环境变量注入。
protocol [string]
连接 Typesense 使用的协议,默认是 http。如果使用 Typesense Cloud 或启用了 TLS 的服务,
请设置为 https。
query [string]
Typesense 查询参数,例如 q=*&filter_by=num_employees:>9000。不配置时读取默认查询返回的文档。
所有合法的 Typesense 搜索参数都可以追加进来,包括 q、query_by、filter_by、sort_by、page 和 per_page,连接器会原样转发给 Typesense 搜索接口。
batch_size [int]
读取数据时每批查询的文档数量。每次请求使用 Typesense 的 per_page 参数,因此该值必须在 1 与 Typesense 服务端 per_page 上限(通常是 250)之间。如果日志中看到分页被截断,请调小该值。
常用选项
Source 插件常用参数,具体请参考 Source 常用选项
任务示例
带过滤条件读取文档
env {
parallelism = 1
job.mode = "BATCH"
}
source {
Typesense {
hosts = ["localhost:8108"]
collection = "companies"
api_key = "xyz"
query = "q=*&filter_by=num_employees:>9000"
batch_size = 100
schema = {
fields {
company_name_list = array<string>
company_name = string
num_employees = long
country = string
id = string
c_row = {
c_int = int
c_string = string
c_array_int = array<int>
}
}
}
}
}
sink {
Console {}
}
使用自定义查询条件读取子集
组合 query_by 与 sort_by 可以控制 Typesense 搜索的字段以及结果集的排序方式。
env {
parallelism = 1
job.mode = "BATCH"
}
source {
Typesense {
hosts = ["localhost:8108"]
collection = "companies"
api_key = "xyz"
query = "q=acme&query_by=company_name&filter_by=country:=US&sort_by=num_employees:desc"
batch_size = 50
schema = {
fields {
company_name = string
num_employees = long
country = string
id = string
}
}
}
}
sink {
Console {}
}
变更日志
Change Log
| Change | Commit | Version |
|---|---|---|
| [Fix][Connector-V2] Add opt-in collision-safe Easysearch and Typesense IDs (#12839) | https://github.com/apache/seatunnel/commit/68c686438f | 3.1.0 |
| [Improve][Connector-V2] Validate Typesense connection options (#12175) | https://github.com/apache/seatunnel/commit/fdb6beb3ff | 3.0.0 |
| [Improve][Connector-V2] Improve source split round-robin assignment for Easysearch, AmazonDynamoDB, TiDB CDC and Typesense (#11607) | https://github.com/apache/seatunnel/commit/9487cc8e16 | 3.0.0 |
| fix typesense add missing import for MULTI_TABLE_SINK_REPLICA to pass… (#10898) | https://github.com/apache/seatunnel/commit/d8e9244a2b | 3.0.0 |
| [Fix][Core] Add shade module for apache commons lang3 (#9895) | https://github.com/apache/seatunnel/commit/abb9124b05 | 2.3.13 |
| [Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671) | https://github.com/apache/seatunnel/commit/9212a77140 | 2.3.12 |
| [improve] typesense options (#9398) | https://github.com/apache/seatunnel/commit/bf20a3e6a8 | 2.3.12 |
| [Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118) | https://github.com/apache/seatunnel/commit/4f5adeb1c7 | 2.3.11 |
| [Fix] Fix error log name for SourceSplitEnumerator implements class (#8817) | https://github.com/apache/seatunnel/commit/55ed90ecaf | 2.3.10 |
| [Improve] restruct connector common options (#8634) | https://github.com/apache/seatunnel/commit/f3499a6eeb | 2.3.10 |
| [Feature]Check Chinese comments in the code (#8319) | https://github.com/apache/seatunnel/commit/d58fce1caf | 2.3.9 |
| [Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef800016 | 2.3.9 |
| [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03c | 2.3.9 |
| [Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700) | https://github.com/apache/seatunnel/commit/e2fb679577 | 2.3.8 |
| [Feature][Connector-V2] Support typesense connector (#7450) | https://github.com/apache/seatunnel/commit/138d2a4eb2 | 2.3.8 |