Hbase
Hbase sink connector
Description
Output data to Hbase
Key features
Options
name | type | required | default value |
---|---|---|---|
zookeeper_quorum | string | yes | - |
table | string | yes | - |
rowkey_column | list | yes | - |
family_name | config | yes | - |
rowkey_delimiter | string | no | "" |
version_column | string | no | - |
null_mode | string | no | skip |
wal_write | boolean | yes | false |
write_buffer_size | string | no | 8 1024 1024 |
encoding | string | no | utf8 |
hbase_extra_config | string | no | - |
common-options | no | - | |
ttl | long | no | - |
zookeeper_quorum [string]
The zookeeper cluster host of hbase, example: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
table [string]
The table name you want to write, example: "seatunnel"
rowkey_column [list]
The column name list of row keys, example: ["id", "uuid"]
family_name [config]
The family name mapping of fields. For example the row from upstream like the following shown:
id | name | age |
---|---|---|
1 | tyrantlucifer | 27 |
id as the row key and other fields written to the different families, you can assign
family_name { name = "info1" age = "info2" }
this means that name
will be written to the family info1
and the age
will be written to the family info2
if you want other fields written to the same family, you can assign
family_name { all_columns = "info" }
this means that all fields will be written to the family info
rowkey_delimiter [string]
The delimiter of joining multi row keys, default ""
version_column [string]
The version column name, you can use it to assign timestamp for hbase record
null_mode [double]
The mode of writing null value, support [skip
, empty
], default skip
- skip: When the field is null, connector will not write this field to hbase
- empty: When the field is null, connector will write generate empty value for this field
wal_write [boolean]
The wal log write flag, default false
write_buffer_size [int]
The write buffer size of hbase client, default 8 * 1024 * 1024
encoding [string]
The encoding of string field, support [utf8
, gbk
], default utf8
hbase_extra_config [config]
The extra configuration of hbase
ttl [long]
Hbase writes data TTL time, the default is based on the TTL set in the table, unit: milliseconds
common options
Sink plugin common parameters, please refer to Sink Common Options for details
Example
Hbase {
zookeeper_quorum = "hadoop001:2181,hadoop002:2181,hadoop003:2181"
table = "seatunnel_test"
rowkey_column = ["name"]
family_name {
all_columns = seatunnel
}
}
Multiple Table
env {
# You can set engine configuration here
execution.parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
tables_configs = [
{
schema = {
table = "hbase_sink_1"
fields {
name = STRING
c_string = STRING
c_double = DOUBLE
c_bigint = BIGINT
c_float = FLOAT
c_int = INT
c_smallint = SMALLINT
c_boolean = BOOLEAN
time = BIGINT
}
}
rows = [
{
kind = INSERT
fields = ["label_1", "sink_1", 4.3, 200, 2.5, 2, 5, true, 1627529632356]
}
]
},
{
schema = {
table = "hbase_sink_2"
fields {
name = STRING
c_string = STRING
c_double = DOUBLE
c_bigint = BIGINT
c_float = FLOAT
c_int = INT
c_smallint = SMALLINT
c_boolean = BOOLEAN
time = BIGINT
}
}
rows = [
{
kind = INSERT
fields = ["label_2", "sink_2", 4.3, 200, 2.5, 2, 5, true, 1627529632357]
}
]
}
]
}
}
sink {
Hbase {
zookeeper_quorum = "hadoop001:2181,hadoop002:2181,hadoop003:2181"
table = "${table_name}"
rowkey_column = ["name"]
family_name {
all_columns = info
}
}
}
Writes To The Specified Column Family
Hbase {
zookeeper_quorum = "hbase_e2e:2181"
table = "assign_cf_table"
rowkey_column = ["id"]
family_name {
c_double = "cf1"
c_bigint = "cf2"
}
}
Changelog
Change Log
Change | Commit | Version |
---|---|---|
[Improve] hbase options (#8923) | https://github.com/apache/seatunnel/commit/b6a702b58 | 2.3.10 |
[Improve] restruct connector common options (#8634) | https://github.com/apache/seatunnel/commit/f3499a6ee | 2.3.10 |
[Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef80001 | 2.3.9 |
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03 | 2.3.9 |
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700) | https://github.com/apache/seatunnel/commit/e2fb67957 | 2.3.8 |
[Feature][Connector-V2][Hbase] implement hbase catalog (#7516) | https://github.com/apache/seatunnel/commit/b978792cb | 2.3.8 |
[Feature][Connector-V2] Support multi-table sink feature for HBase (#7169) | https://github.com/apache/seatunnel/commit/025fa3bb8 | 2.3.8 |
[hotfix][connector-v2-hbase]fix and optimize hbase source problem (#7148) | https://github.com/apache/seatunnel/commit/34a6b8e9f | 2.3.7 |
[Improve][hbase] The specified column is written to the specified column family (#5234) | https://github.com/apache/seatunnel/commit/49d397c61 | 2.3.6 |
[feature][connector-v2-hbase-sink] Support Connector v2 HBase sink TTL data writing (#7116) | https://github.com/apache/seatunnel/commit/adafd8025 | 2.3.6 |
[E2E][HBase]Refactor hbase e2e (#6859) | https://github.com/apache/seatunnel/commit/1da9bd6ce | 2.3.6 |
[Connector]Add hbase source connector (#6348) | https://github.com/apache/seatunnel/commit/f108a5e65 | 2.3.6 |
[Feature][HbaseSink]support array data. (#6100) | https://github.com/apache/seatunnel/commit/b59201476 | 2.3.4 |
[Improve][Common] Introduce new error define rule (#5793) | https://github.com/apache/seatunnel/commit/9d1b2582b | 2.3.4 |
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755) | https://github.com/apache/seatunnel/commit/8de740810 | 2.3.4 |
[Hotfix][Connector-v2][HbaseSink]Fix default timestamp (#4958) | https://github.com/apache/seatunnel/commit/3d8f3bf90 | 2.3.3 |
[Improve][build] Give the maven module a human readable name (#4114) | https://github.com/apache/seatunnel/commit/d7cd60105 | 2.3.1 |
[Improve][Project] Code format with spotless plugin. (#4101) | https://github.com/apache/seatunnel/commit/a2ab16656 | 2.3.1 |
[Feature][Connector-V2][Hbase] Introduce hbase sink connector (#4049) | https://github.com/apache/seatunnel/commit/68bda94a4 | 2.3.1 |