跳到主要内容
版本:Next

使用Docker进行部署

使用Docker启用本地模式

Zeta 引擎

下载镜像

docker pull apache/seatunnel:<version_tag>

当下载完成后,可以使用如下命令来提交任务

# Run fake source to console sink
docker run --rm -it apache/seatunnel:<version_tag> ./bin/seatunnel.sh -m local -c config/v2.batch.config.template

# Run job with custom config file
docker run --rm -it -v /<The-Config-Directory-To-Mount>/:/config apache/seatunnel:<version_tag> ./bin/seatunnel.sh -m local -c /config/fake_to_console.conf

# Example
# If you config file is in /tmp/job/fake_to_console.conf
docker run --rm -it -v /tmp/job/:/config apache/seatunnel:<version_tag> ./bin/seatunnel.sh -m local -c /config/fake_to_console.conf

# Set JVM options when running
docker run --rm -it -v /tmp/job/:/config apache/seatunnel:<version_tag> ./bin/seatunnel.sh -DJvmOption="-Xms4G -Xmx4G" -m local -c /config/fake_to_console.conf

自己构建镜像

从源代码构建。下载源码的方式和下载二进制包的方式是一样的。 你可以从下载地址下载源码, 或者从GitHub 仓库克隆源代码

一个命令来构建容器
cd seatunnel
# Use already sett maven profile
mvn -B clean install -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dlicense.skipAddThirdParty=true -D"docker.build.skip"=false -D"docker.verify.skip"=false -D"docker.push.skip"=true -D"docker.tag"=2.3.9 -Dmaven.deploy.skip -D"skip.spotless"=true --no-snapshot-updates -Pdocker,seatunnel

# Check the docker image
docker images | grep apache/seatunnel
分步骤构建
# Build binary package from source code
mvn clean package -DskipTests -Dskip.spotless=true

# Build docker image
cd seatunnel-dist
docker build -f src/main/docker/Dockerfile --build-arg VERSION=2.3.9 -t apache/seatunnel:2.3.9 .

# If you build from dev branch, you should add SNAPSHOT suffix to the version
docker build -f src/main/docker/Dockerfile --build-arg VERSION=2.3.9-SNAPSHOT -t apache/seatunnel:2.3.9-SNAPSHOT .

# Check the docker image
docker images | grep apache/seatunnel

Dockerfile文件内容为:

FROM openjdk:8

ARG VERSION
# Build from Source Code And Copy it into image
COPY ./target/apache-seatunnel-${VERSION}-bin.tar.gz /opt/

# Download From Internet
# Please Note this file only include fake/console connector, You'll need to download the other connectors manually
# wget -P /opt https://dlcdn.apache.org/seatunnel/2.3.6/apache-seatunnel-${VERSION}-bin.tar.gz

RUN cd /opt && \
tar -zxvf apache-seatunnel-${VERSION}-bin.tar.gz && \
mv apache-seatunnel-${VERSION} seatunnel && \
rm apache-seatunnel-${VERSION}-bin.tar.gz && \
cp seatunnel/config/log4j2_client.properties seatunnel/config/log4j2.properties && \
cp seatunnel/config/hazelcast-master.yaml seatunnel/config/hazelcast-worker.yaml

WORKDIR /opt/seatunnel

Spark/Flink引擎

默认设值下,Spark的目录为/opt/spark, Flink的目录为 /opt/flink. 如果你需要运行Spark或Flink引擎,你需要将相关依赖挂载到/opt/spark/opt/flink目录下.

docker run \ 
-v <SPARK_BINARY_PATH>:/opt/spark \
-v <FLINK_BINARY_PATH>:/opt/flink \
...

或者你可以在Dockerfile中修改 SPARK_HOME, FLINK_HOME环境变量,并且重新构建基础镜像,然后再进行挂载.

FROM apache/seatunnel

ENV SPARK_HOME=<YOUR_CUSTOMIZATION_PATH>

...

docker run \ 
-v <SPARK_BINARY_PATH>:<YOUR_CUSTOMIZATION_PATH> \
...

提交任务

不同引擎和同一引擎的不同版本命令不同,请选择正确的命令。

  • Spark
# spark2
docker run --rm -it apache/seatunnel bash ./bin/start-seatunnel-spark-2-connector-v2.sh -c config/v2.batch.config.template

# spark3
docker run --rm -it apache/seatunnel bash ./bin/start-seatunnel-spark-3-connector-v2.sh -c config/v2.batch.config.template
  • Flink 在提交作业之前,您需要先启动 Flink 集群。
# flink version between `1.12.x` and `1.14.x`
docker run --rm -it apache/seatunnel bash -c '<YOUR_FLINK_HOME>/bin/start-cluster.sh && ./bin/start-seatunnel-flink-13-connector-v2.sh -c config/v2.streaming.conf.template'
# flink version between `1.15.x` and `1.16.x`
docker run --rm -it apache/seatunnel bash -c '<YOUR_FLINK_HOME>/bin/start-cluster.sh && ./bin/start-seatunnel-flink-15-connector-v2.sh -c config/v2.streaming.conf.template'

使用Docker配置集群模式

docker下的集群模式仅支持Zeta引擎

有两种方式来启动集群

直接使用Docker

创建一个network

docker network create seatunnel-network

启动节点

  • 启动master节点
## start master and export 5801 port 
docker run -d --name seatunnel_master \
--network seatunnel-network \
--rm \
-p 5801:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r master
  • 获取容器的ip
docker inspect seatunnel_master

运行此命令获取master容器的ip

  • 启动worker节点
# 将ST_DOCKER_MEMBER_LIST设置为master容器的ip
docker run -d --name seatunnel_worker_1 \
--network seatunnel-network \
--rm \
-e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r worker

## 启动第二个worker节点
# 将ST_DOCKER_MEMBER_LIST设置为master容器的ip
docker run -d --name seatunnel_worker_2 \
--network seatunnel-network \
--rm \
-e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r worker

集群扩容

# 将ST_DOCKER_MEMBER_LIST设置为已经启动的master容器的ip 
docker run -d --name seatunnel_master \
--network seatunnel-network \
--rm \
-e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r master

运行这个命令创建一个worker节点

# 将ST_DOCKER_MEMBER_LIST设置为master容器的ip
docker run -d --name seatunnel_worker_1 \
--network seatunnel-network \
--rm \
-e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
apache/seatunnel \
./bin/seatunnel-cluster.sh -r worker

使用docker-compose

docker-compose.yaml 配置文件为:

version: '3.8'

services:
master:
image: apache/seatunnel
container_name: seatunnel_master
environment:
- ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4
entrypoint: >
/bin/sh -c "
/opt/seatunnel/bin/seatunnel-cluster.sh -r master
"
ports:
- "5801:5801"
networks:
seatunnel_network:
ipv4_address: 172.16.0.2

worker1:
image: apache/seatunnel
container_name: seatunnel_worker_1
environment:
- ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4
entrypoint: >
/bin/sh -c "
/opt/seatunnel/bin/seatunnel-cluster.sh -r worker
"
depends_on:
- master
networks:
seatunnel_network:
ipv4_address: 172.16.0.3

worker2:
image: apache/seatunnel
container_name: seatunnel_worker_2
environment:
- ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4
entrypoint: >
/bin/sh -c "
/opt/seatunnel/bin/seatunnel-cluster.sh -r worker
"
depends_on:
- master
networks:
seatunnel_network:
ipv4_address: 172.16.0.4

networks:
seatunnel_network:
driver: bridge
ipam:
config:
- subnet: 172.16.0.0/24

运行 docker-compose up命令来启动集群,该配置会启动一个master节点,2个worker节点

启动完成后,可以运行docker logs -f seatunne_master, docker logs -f seatunnel_worker_1来查看节点的日志
当你访问http://localhost:5801/hazelcast/rest/maps/system-monitoring-information 时,可以看到集群的状态为1个master节点,2个worker节点.

集群扩容

当你需要对集群扩容, 例如需要添加一个worker节点时

version: '3.8'

services:
master:
image: apache/seatunnel
container_name: seatunnel_master
environment:
- ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4
entrypoint: >
/bin/sh -c "
/opt/seatunnel/bin/seatunnel-cluster.sh -r master
"
ports:
- "5801:5801"
networks:
seatunnel_network:
ipv4_address: 172.16.0.2

worker1:
image: apache/seatunnel
container_name: seatunnel_worker_1
environment:
- ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4
entrypoint: >
/bin/sh -c "
/opt/seatunnel/bin/seatunnel-cluster.sh -r worker
"
depends_on:
- master
networks:
seatunnel_network:
ipv4_address: 172.16.0.3

worker2:
image: apache/seatunnel
container_name: seatunnel_worker_2
environment:
- ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4
entrypoint: >
/bin/sh -c "
/opt/seatunnel/bin/seatunnel-cluster.sh -r worker
"
depends_on:
- master
networks:
seatunnel_network:
ipv4_address: 172.16.0.4
####
## 添加新节点配置
####
worker3:
image: apache/seatunnel
container_name: seatunnel_worker_3
environment:
- ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4,172.16.0.5 # 添加ip到这里
entrypoint: >
/bin/sh -c "
/opt/seatunnel/bin/seatunnel-cluster.sh -r worker
"
depends_on:
- master
networks:
seatunnel_network:
ipv4_address: 172.16.0.5 # 设置新节点ip

networks:
seatunnel_network:
driver: bridge
ipam:
config:
- subnet: 172.16.0.0/24

然后运行docker-compose up -d命令, 将会新建一个worker节点, 已有的节点不会重启.

提交作业到集群

使用docker container作为客户端

  • 提交任务
# 将ST_DOCKER_MEMBER_LIST设置为master容器的ip
docker run --name seatunnel_client \
--network seatunnel-network \
-e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
--rm \
apache/seatunnel \
./bin/seatunnel.sh -c config/v2.batch.config.template
  • 查看作业列表
# 将ST_DOCKER_MEMBER_LIST设置为master容器的ip
docker run --name seatunnel_client \
--network seatunnel-network \
-e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 \
--rm \
apache/seatunnel \
./bin/seatunnel.sh -l

更多其他命令请参考命令行工具

使用RestAPI

请参考 提交作业