Common Options

Common parameters of transform plugins

tip

This transform both supported by engine Spark and Flink.

Transform Plugin common parameters

Spark

name	type	required	default value
source_table_name	string	no	-
result_table_name	string	no	-

Flink

name	type	required	default value
source_table_name	string	no	-
result_table_name	string	no	-
field_name	string	no	-

field_name [string]

When the data is obtained from the upper-level plugin, you can specify the name of the obtained field, which is convenient for use in subsequent sql plugins.

source_table_name [string]

When source_table_name is not specified, the current plug-in processes the data set (dataset) output by the previous plug-in in the configuration file;

When source_table_name is specified, the current plugin is processing the data set corresponding to this parameter.

result_table_name [string]

When result_table_name is not specified, the data processed by this plugin will not be registered as a data set that can be directly accessed by other plugins, or called a temporary table (table);

When result_table_name is specified, the data processed by this plugin will be registered as a data set (dataset) that can be directly accessed by other plugins, or called a temporary table (table) . The dataset registered here can be directly accessed by other plugins by specifying source_table_name .

Examples

Spark

split {
    source_table_name = "source_view_table"
    source_field = "message"
    delimiter = "&"
    fields = ["field1", "field2"]
    result_table_name = "result_view_table"
}

The Split plugin will process the data in the temporary table source_view_table and register the processing result as a temporary table named result_view_table. This temporary table can be used by any subsequent Filter or Output plugin by specifying source_table_name .

split {
    source_field = "message"
    delimiter = "&"
    fields = ["field1", "field2"]
}

Note: If source_table_name is not configured, output the processing result of the last Transform plugin in the configuration file

Flink

source {
    FakeSourceStream {
      result_table_name = "fake_1"
      field_name = "name,age"
    }
    FakeSourceStream {
      result_table_name = "fake_2"
      field_name = "name,age"
    }
}

transform {
    sql {
      source_table_name = "fake_1"
      sql = "select name from fake_1"
      result_table_name = "fake_name"
    }
}

If source_table_name is not specified, the sql plugin will process the data of fake_2 , and if it is set to fake_1 , it will process the data of fake_1 .