Elasticsearch
Elasticsearch source connector
Description
Used to read data from Elasticsearch.
support version >= 2.x and <= 8.x.
Key features
Options
name | type | required | default value |
---|---|---|---|
hosts | array | yes | - |
username | string | no | - |
password | string | no | - |
index | string | yes | - |
source | array | no | - |
query | json | no | {"match_all": {}} |
scroll_time | string | no | 1m |
scroll_size | int | no | 100 |
tls_verify_certificate | boolean | no | true |
tls_verify_hostnames | boolean | no | true |
array_column | map | no | |
tls_keystore_path | string | no | - |
tls_keystore_password | string | no | - |
tls_truststore_path | string | no | - |
tls_truststore_password | string | no | - |
common-options | no | - |
hosts [array]
Elasticsearch cluster http address, the format is host:port
, allowing multiple hosts to be specified. Such as ["host1:9200", "host2:9200"]
.
username [string]
x-pack username.
password [string]
x-pack password.
index [string]
Elasticsearch index name, support * fuzzy matching.
source [array]
The fields of index.
You can get the document id by specifying the field _id
.If sink _id to other index,you need specify an alias for _id due to the Elasticsearch limit.
If you don't config source, it is automatically retrieved from the mapping of the index.
array_column [array]
The fields of array type.
Since there is no array index in es,so need assign array type,just like {c_array = "array<tinyint>"}
.
query [json]
Elasticsearch DSL. You can control the range of data read.
scroll_time [String]
Amount of time Elasticsearch will keep the search context alive for scroll requests.
scroll_size [int]
Maximum number of hits to be returned with each Elasticsearch scroll request.
tls_verify_certificate [boolean]
Enable certificates validation for HTTPS endpoints
tls_verify_hostname [boolean]
Enable hostname validation for HTTPS endpoints
tls_keystore_path [string]
The path to the PEM or JKS key store. This file must be readable by the operating system user running SeaTunnel.
tls_keystore_password [string]
The key password for the key store specified
tls_truststore_path [string]
The path to PEM or JKS trust store. This file must be readable by the operating system user running SeaTunnel.
tls_truststore_password [string]
The key password for the trust store specified
common options
Source plugin common parameters, please refer to Source Common Options for details
Examples
simple
Elasticsearch {
hosts = ["localhost:9200"]
index = "seatunnel-*"
source = ["_id","name","age"]
query = {"range":{"firstPacket":{"gte":1669225429990,"lte":1669225429990}}}
}
complex
Elasticsearch {
hosts = ["elasticsearch:9200"]
index = "st_index"
schema = {
fields {
c_map = "map<string, tinyint>"
c_array = "array<tinyint>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_decimal = "decimal(2, 1)"
c_bytes = bytes
c_date = date
c_timestamp = timestamp
}
}
query = {"range":{"firstPacket":{"gte":1669225429990,"lte":1669225429990}}}
}
SSL (Disable certificates validation)
source {
Elasticsearch {
hosts = ["https://localhost:9200"]
username = "elastic"
password = "elasticsearch"
tls_verify_certificate = false
}
}
SSL (Disable hostname validation)
source {
Elasticsearch {
hosts = ["https://localhost:9200"]
username = "elastic"
password = "elasticsearch"
tls_verify_hostname = false
}
}
SSL (Enable certificates validation)
source {
Elasticsearch {
hosts = ["https://localhost:9200"]
username = "elastic"
password = "elasticsearch"
tls_keystore_path = "${your elasticsearch home}/config/certs/http.p12"
tls_keystore_password = "${your password}"
}
}
Changelog
next version
- Add Elasticsearch Source Connector
- [Feature] Support https protocol & compatible with opensearch (3997)
- [Feature] Support DSL