Easysearch

Easysearch source connector

Support Those Engines

Spark
Flink
SeaTunnel Zeta

Description

Used to read data from INFINI Easysearch.

Using Dependency

Depenndency easysearch-client

Key features

• batch
• stream
• exactly-once
• column projection
• parallelism
• support user-defined split

tip

Engine Supported

• Supported all versions released by INFINI Easysearch.

Data Type Mapping

Easysearch Data Type	SeaTunnel Data Type
STRING KEYWORD TEXT	STRING
BOOLEAN	BOOLEAN
BYTE	BYTE
SHORT	SHORT
INTEGER	INT
LONG	LONG
FLOAT HALF_FLOAT	FLOAT
DOUBLE	DOUBLE
Date	LOCAL_DATE_TIME_TYPE

hosts [array]

Easysearch cluster http address, the format is host:port, allowing multiple hosts to be specified. Such as ["host1:9200", "host2:9200"].

username [string]

security username.

password [string]

security password.

index [string]

Easysearch index name, support * fuzzy matching.

source [array]

The fields of index. You can get the document id by specifying the field _id.If sink _id to other index,you need specify an alias for _id due to the Easysearch limit. If you don't config source, you must config schema.

query [json]

Easysearch DSL. You can control the range of data read.

scroll_time [String]

Amount of time Easysearch will keep the search context alive for scroll requests.

scroll_size [int]

Maximum number of hits to be returned with each Easysearch scroll request.

schema

The structure of the data, including field names and field types. If you don't config schema, you must config source.

tls_verify_certificate [boolean]

Enable certificates validation for HTTPS endpoints

tls_verify_hostname [boolean]

Enable hostname validation for HTTPS endpoints

tls_keystore_path [string]

The path to the PEM or JKS key store. This file must be readable by the operating system user running SeaTunnel.

tls_keystore_password [string]

The key password for the key store specified

tls_truststore_path [string]

The path to PEM or JKS trust store. This file must be readable by the operating system user running SeaTunnel.

tls_truststore_password [string]

The key password for the trust store specified

common options

Source plugin common parameters, please refer to Source Common Options for details

Examples

simple

Easysearch {
    hosts = ["localhost:9200"]
    index = "seatunnel-*"
    source = ["_id","name","age"]
    query = {"range":{"firstPacket":{"gte":1700407367588,"lte":1700407367588}}}
}

complex

Easysearch {
    hosts = ["Easysearch:9200"]
    index = "st_index"
    schema = {
        fields {
            c_map = "map<string, tinyint>"
            c_array = "array<tinyint>"
            c_string = string
            c_boolean = boolean
            c_tinyint = tinyint
            c_smallint = smallint
            c_int = int
            c_bigint = bigint
            c_float = float
            c_double = double
            c_decimal = "decimal(2, 1)"
            c_bytes = bytes
            c_date = date
            c_timestamp = timestamp
        }
    }
    query = {"range":{"firstPacket":{"gte":1700407367588,"lte":1700407367588}}}
}

SSL (Disable certificates validation)

source {
    Easysearch {
        hosts = ["https://localhost:9200"]
        username = "admin"
        password = "admin"
        
        tls_verify_certificate = false
    }
}

SSL (Disable hostname validation)

source {
    Easysearch {
        hosts = ["https://localhost:9200"]
        username = "admin"
        password = "admin"
        
        tls_verify_hostname = false
    }
}

SSL (Enable certificates validation)

source {
    Easysearch {
        hosts = ["https://localhost:9200"]
        username = "admin"
        password = "admin"
        
        tls_keystore_path = "${your Easysearch home}/config/certs/http.p12"
        tls_keystore_password = "${your password}"
    }
}

Changelog

Change Log

Change	Commit	Version
[improve] EasySearch support schema_save_mode/data_save_mode (#9310)	https://github.com/apache/seatunnel/commit/3ceb57f279	2.3.11
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)	https://github.com/apache/seatunnel/commit/4f5adeb1c7	2.3.11
[Improve] easysearch options (#8951)	https://github.com/apache/seatunnel/commit/349f142962	2.3.10
[Fix] Fix error log name for SourceSplitEnumerator implements class (#8817)	https://github.com/apache/seatunnel/commit/55ed90ecaf	2.3.10
[Improve] restruct connector common options (#8634)	https://github.com/apache/seatunnel/commit/f3499a6eeb	2.3.10
[Improve][dist]add shade check rule (#8136)	https://github.com/apache/seatunnel/commit/51ef800016	2.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)	https://github.com/apache/seatunnel/commit/6b7c53d03c	2.3.9
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700)	https://github.com/apache/seatunnel/commit/e2fb679577	2.3.8
[Feature][Doris] Add Doris type converter (#6354)	https://github.com/apache/seatunnel/commit/5189991843	2.3.6
[Hotfix] Fix compile error (#6463)	https://github.com/apache/seatunnel/commit/943bd48449	2.3.5
[Improve][Connector-V2] Support INFINI Easysearch (#5933)	https://github.com/apache/seatunnel/commit/41e628840a	2.3.5