Skip to main content
Version: Next

Tablestore

Tablestore source connector

Description​

Read full and incremental data from Alibaba Cloud Tablestore. The source uses Tablestore Tunnel in BaseAndStream mode, so it can read existing data first and then consume later changes.

Key features​

Options​

nametyperequireddefault valuedescription
end_pointstringyes-Tablestore endpoint, for example https://<instance>.<region>.ots.aliyuncs.com.
instance_namestringyes-Tablestore instance name.
access_key_idstringyes-AccessKey ID used to access Tablestore.
access_key_secretstringyes-AccessKey secret used to access Tablestore.
tablestringyes-Tablestore table name. Multiple tables can be separated by commas.
primary_keysarrayyes-Primary key names. For multiple source tables, configure one primary key name for each table in the same order as table.
schemaconfigyes-Output schema. For details, see Schema Feature.

Usage notes​

  • job.mode = "BATCH" reads bounded data. job.mode = "STREAMING" keeps consuming incremental records after the existing data is read.
  • When table contains multiple table names, primary_keys must contain the same number of entries. For example, table = "orders,users" can use primary_keys = ["id", "id"] when both tables use id as the primary key field.
  • Multi-table reads use one schema block for the source, so the listed tables should have compatible output fields.
  • The source emits INSERT, UPDATE_AFTER, and DELETE row kinds according to Tablestore stream records.
  • Keep access_key_id and access_key_secret out of committed job files. Prefer runtime variable substitution or a secret manager supported by your deployment environment.

Example​

env {
  parallelism = 1
  job.mode = "STREAMING"
}

source {
  Tablestore {
    end_point = "https://<instance>.<region>.ots.aliyuncs.com"
    instance_name = "<instance-name>"
    access_key_id = "${ACCESS_KEY_ID}"
    access_key_secret = "${ACCESS_KEY_SECRET}"
    table = "orders"
    primary_keys = ["order_id"]
    schema = {
      fields {
        order_id = string
        user_id = string
        amount = double
        updated_at = string
      }
    }
  }
}

sink {
  Console {}
}

Multiple table example​

env {
  parallelism = 2
  job.mode = "STREAMING"
}

source {
  Tablestore {
    end_point = "https://<instance>.<region>.ots.aliyuncs.com"
    instance_name = "<instance-name>"
    access_key_id = "${ACCESS_KEY_ID}"
    access_key_secret = "${ACCESS_KEY_SECRET}"
    table = "orders,users"
    primary_keys = ["id", "id"]
    schema = {
      fields {
        id = string
        value = string
        updated_at = string
      }
    }
  }
}

sink {
  Console {}
}

Changelog​

Change Log
ChangeCommitVersion
[Improve] table_store options (#9515)https://github.com/apache/seatunnel/commit/145b68793f2.3.12
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Feature][Connector-V2][Tablestore] Support Source connector for Tablestore #7448 (#7467)https://github.com/apache/seatunnel/commit/a7ca51b5852.3.8
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
Support config column/primaryKey/constraintKey in schema (#5564)https://github.com/apache/seatunnel/commit/eac76b4e502.3.4
[Improve][Connector-V2] Remove scheduler in Tablestore sink (#5272)https://github.com/apache/seatunnel/commit/8d6b07e4662.3.3
Merge branch 'dev' into merge/cdchttps://github.com/apache/seatunnel/commit/4324ee19122.3.1
[Improve][Project] Code format with spotless plugin.https://github.com/apache/seatunnel/commit/423b5830382.3.1
[improve][api] Refactoring schema parse (#4157)https://github.com/apache/seatunnel/commit/b2f573a13e2.3.1
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Hotfix][OptionRule] Fix option rule about all connectors (#3592)https://github.com/apache/seatunnel/commit/226dc6a1192.3.0
[Improve][Connector-V2][TableStore] Unified excetion for TableStore sink connector (#3527)https://github.com/apache/seatunnel/commit/7b264d70042.3.0
[Feature][connector-v2] add tablestore source and sink (#3309)https://github.com/apache/seatunnel/commit/ebebf0b6332.3.0