Version: Next

Config File Encryption And Decryption

Introduction

In most production environments, sensitive configuration items such as passwords are required to be encrypted and cannot be stored in plain text, SeaTunnel provides a convenient one-stop solution for this.

How to use

SeaTunnel comes with the function of base64 encryption and decryption, but it is not recommended for production use, it is recommended that users implement custom encryption and decryption logic. You can refer to this chapter [How to implement user-defined encryption and decryption](#How to implement user-defined encryption and decryption) get more details about it.

Base64 encryption support encrypt the following parameters:

username
password
auth

Next, I'll show how to quickly use SeaTunnel's own base64 encryption:

And a new option shade.identifier in env block of config file, this option indicate what the encryption method that you want to use, in this example, we should add shade.identifier = base64 in config as the following shown:

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

env {
  parallelism = 1
  shade.identifier = "base64"
}

source {
  MySQL-CDC {
    plugin_output = "fake"
    parallelism = 1
    server-id = 5656
    port = 56725
    hostname = "127.0.0.1"
    username = "seatunnel"
    password = "seatunnel_password"
    database-name = "inventory_vwyw0n"
    table-name = "products"
    base-url = "jdbc:mysql://localhost:56725"
  }
}

transform {
}

sink {
  # choose stdout output plugin to output data to console
  Clickhouse {
    host = "localhost:8123"
    database = "default"
    table = "fake_all"
    username = "seatunnel"
    password = "seatunnel_password"

    # cdc options
    primary_key = "id"
    support_upsert = true
  }
}

Using the shell based on different calculate engine to encrypt config file, in this example we use zeta:

${SEATUNNEL_HOME}/bin/seatunnel.sh --config config/v2.batch.template --encrypt

Then you can see the encrypted configuration file in the terminal:

2023-02-20 17:50:58,319 INFO  org.apache.seatunnel.core.starter.command.ConfEncryptCommand - Encrypt config: 
{
    "env" : {
        "parallelism" : 1,
        "shade.identifier" : "base64"
    },
    "source" : [
        {
            "base-url" : "jdbc:mysql://localhost:56725",
            "hostname" : "127.0.0.1",
            "password" : "c2VhdHVubmVsX3Bhc3N3b3Jk",
            "port" : 56725,
            "database-name" : "inventory_vwyw0n",
            "parallelism" : 1,
            "plugin_output" : "fake",
            "table-name" : "products",
            "plugin_name" : "MySQL-CDC",
            "server-id" : 5656,
            "username" : "c2VhdHVubmVs"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "database" : "default",
            "password" : "c2VhdHVubmVsX3Bhc3N3b3Jk",
            "support_upsert" : true,
            "host" : "localhost:8123",
            "plugin_name" : "Clickhouse",
            "primary_key" : "id",
            "table" : "fake_all",
            "username" : "c2VhdHVubmVs"
        }
    ]
}

Of course, not only encrypted configuration files are supported, but if the user wants to see the decrypted configuration file, you can execute this command:
```
${SEATUNNEL_HOME}/bin/seatunnel.sh --config config/v2.batch.template --decrypt
```

How to implement user-defined encryption and decryption

If you want to customize the encryption method and the configuration of the encryption, this section will help you to solve the problem.

Create a java maven project

Add seatunnel-api module in dependencies like the following shown:

<dependency>
    <groupId>org.apache.seatunnel</groupId>
    <artifactId>seatunnel-api</artifactId>
    <version>${seatunnel.version}</version>
</dependency>

Create a new class and implement interface ConfigShade, this interface has the following methods:

/**
 * The interface that provides the ability to encrypt and decrypt {@link
 * org.apache.seatunnel.shade.com.typesafe.config.Config}
 */
public interface ConfigShade {

    /**
     * The unique identifier of the current interface, used it to select the correct {@link
     * ConfigShade}
     */
    String getIdentifier();

    /**
     * Encrypt the content
     *
     * @param content The content to encrypt
     */
    String encrypt(String content);

    /**
     * Decrypt the content
     *
     * @param content The content to decrypt
     */
    String decrypt(String content);

    /** To expand the options that user want to encrypt */
    default String[] sensitiveOptions() {
        return new String[0];
    }
}

Add org.apache.seatunnel.api.configuration.ConfigShade in resources/META-INF/services
Package it to jar and add jar to ${SEATUNNEL_HOME}/lib
Change the option shade.identifier to the value that you defined in ConfigShade#getIdentifierof you config file, please enjoy it \^_\^

Config File Encryption And Decryption

Introduction​

How to use​

How to implement user-defined encryption and decryption​

Introduction

How to use

How to implement user-defined encryption and decryption