Sls

Sls sink connector

Support Those Engines

Spark
Flink
SeaTunnel Zeta

Key Features

• exactly-once
• cdc
• timer flush

Description

The Sls sink connector writes SeaTunnel rows to Alibaba Cloud Simple Log Service (SLS). Each SeaTunnel row is serialized as JSON and written to SLS as a log item whose content key is content.

Supported DataSource Info

To use the Sls connector, download the following dependency by using install-plugin.sh or from the Maven central repository.

Datasource	Supported Versions	Maven
Sls	Universal	Download

Sink Options

Name	Type	Required	Default	Description
endpoint	String	Yes	-	Alibaba Cloud SLS endpoint, for example cn-hangzhou.log.aliyuncs.com or an intranet endpoint.
project	String	Yes	-	Alibaba Cloud SLS project.
logstore	String	Yes	-	Alibaba Cloud SLS logstore.
access_key_id	String	Yes	-	Alibaba Cloud AccessKey ID.
access_key_secret	String	Yes	-	Alibaba Cloud AccessKey secret.
source	String	No	SeaTunnel-Source	Source tag written to SLS log groups.
topic	String	No	SeaTunnel-Topic	Topic tag written to SLS log groups.

Notes

• The configured RAM user must have permission to write logs to the target project and logstore.
• The sink writes data as soon as write is called. It does not provide exactly-once commit semantics. In streaming mode the connector flushes row by row; rely on checkpointing only for downstream state, not for the SLS writes themselves.
• Each row is serialized as a JSON object and stored under the content key of an SLS log item. The remaining row fields are not mapped to separate log keys.
• Do not print access_key_secret in logs or job descriptions.

Task Example

Write Rows to SLS (Batch)

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  FakeSource {
    row.num = 10
    map.size = 10
    array.size = 10
    bytes.length = 10
    string.length = 10
    schema = {
      fields = {
        id = "int"
        name = "string"
        description = "string"
        weight = "string"
      }
    }
  }
}

sink {
  Sls {
    endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
    project = "project1"
    logstore = "logstore1"
    access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
    access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    source = "seatunnel-demo"
    topic = "fake-source"
  }
}

Write Rows to SLS (Streaming)

In streaming mode the connector keeps the SLS producer connection open and writes each row as it arrives. Configure checkpoint.interval to make downstream state recoverable, but keep in mind that each PutLogs call is independent and may retry only within the open client session.

env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 30000
}

source {
  FakeSource {
    row.num = 10
    map.size = 10
    array.size = 10
    bytes.length = 10
    string.length = 10
    schema = {
      fields = {
        id = "int"
        name = "string"
        description = "string"
        weight = "string"
      }
    }
  }
}

sink {
  Sls {
    endpoint = "cn-hangzhou.log.aliyuncs.com"
    project = "project1"
    logstore = "logstore1"
    access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
    access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    source = "seatunnel-streaming"
    topic = "fake-source"
  }
}

Changelog

Change Log

Change	Commit	Version
[Chore] fix typos filed -> field (#9757)	https://github.com/apache/seatunnel/commit/e3e1c67d29	2.3.12
[Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671)	https://github.com/apache/seatunnel/commit/9212a77140	2.3.12
[improve] sls options (#9260)	https://github.com/apache/seatunnel/commit/126164508b	2.3.11
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)	https://github.com/apache/seatunnel/commit/4f5adeb1c7	2.3.11
[Improve] restruct connector common options (#8634)	https://github.com/apache/seatunnel/commit/f3499a6eeb	2.3.10
[Feature]Check Chinese comments in the code (#8319)	https://github.com/apache/seatunnel/commit/d58fce1caf	2.3.9
[Improve][dist]add shade check rule (#8136)	https://github.com/apache/seatunnel/commit/51ef800016	2.3.9
[Improve][Sls] Add sls sink connector、e2e、doc (#7830)	https://github.com/apache/seatunnel/commit/048c47d966	2.3.9
[Fix][Connector-V2] Fix some throwable error not be caught (#7657)	https://github.com/apache/seatunnel/commit/e19d73282e	2.3.8
[Feature][Connector-V2] add Aliyun SLS connector #3733 (#7348)	https://github.com/apache/seatunnel/commit/527c7c7b5f	2.3.7