TableMerge
TableMerge transform plugin
Description
TableMerge transform plugin for merge sharding-tables.
Options
| name | type | required | default value | Description |
|---|---|---|---|---|
| database | string | no | Specify new database name | |
| schema | string | no | Specify new schema name | |
| table | string | conditional | Required only when table_transform is unset or empty; each table_transform entry still requires its own table. | |
| table_match_regex | string | no | .* | Regex of the table paths that the transform level rule applies to. |
| table_transform | array | no | [] | Per-table rules, see Multi-Table Transform. Exact table-path entries take precedence over the regex; an entry with only table_match_regex does not instantiate the global rule. |
| rule_match_mode | string | no | FIRST_MATCH | How multiple table_transform entries with the same table_path are evaluated. Omitting it rejects duplicate entries; FIRST_MATCH selects the first one. |
Tables that match neither table_match_regex nor a table_transform entry are passed through unchanged. In the example below, source.shop does not match source.user_.*, so it keeps its original name while source.user_1 and source.user_2 are merged into user_db.user_all.
Examples
Merge sharding-tables
env {
parallelism = 1
job.mode = "STREAMING"
}
source {
MySQL-CDC {
plugin_output = "customers_mysql_cdc"
username = "root"
password = "123456"
table-names = ["source.user_1", "source.user_2", "source.shop"]
url = "jdbc:mysql://localhost:3306/source"
}
}
transform {
TableMerge {
plugin_input = "customers_mysql_cdc"
plugin_output = "trans_result"
table_match_regex = "source.user_.*"
database = "user_db"
table = "user_all"
}
}
sink {
Jdbc {
plugin_input = "trans_result"
driver="com.mysql.cj.jdbc.Driver"
url="jdbc:mysql://localhost:3306/sink"
user="myuser"
password="mypwd"
generate_sink_sql = true
database = "${database_name}"
table = "${table_name}"
primary_keys = ["${primary_key}"]
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
data_save_mode = "APPEND_DATA"
}
}