SeaTunnel-web in K8S

crabdave123 2024-07-27 13:33:02 阅读 68

下载,官网下载有问题,上dlcdn.apache.org下载

https://dlcdn.apache.org/seatunnel/seatunnel-web/1.0.0/apache-seatunnel-web-1.0.0-bin.tar.gz

apache-seatunnel-2.3.3中执行bin/install-plugin.sh下载connectors

下载web的源码

https://github.com/apache/seatunnel-web.git

seatunnel-server/seatunnel-app/src/main/bin/download_datasource.sh

将download_datasource.sh复制到/root/seatunnel/apache-seatunnel-web-1.0.0-bin/bin目录

sh download_datasource.sh 下载 datasource

vim conf/application.yml修改端口号和数据源信息

根据script目录下的数据表初始化脚本,把数据库建好 seatunnel_server_mysql.sql

将apache-seatunnel-2.3.4/conf目录下的hazelcast-client.yamlplugin-mapping.properties复制过来

hazelcast-client.yaml中的信息连接到之前部署好的SeaTunnel集群

参考:SeaTunnel 2.3.4 Cluster in K8S-CSDN博客

logback-spring.xml文件中的日志级别如果需要可改成debug方便调试

准备创建镜像文件

apache-seatunnel-2.3.3(重命名为:seatunnel-zeta)和apache-seatunnel-web-1.0.0-bin(重命名为:seatunnel,web依赖zeta连接集群)都放在一个容器中。

在镜像中配置相关的环境变量

SEATUNNEL_HOME=/opt/seatunnel-zeta

SEATUNNEL_WEB_HOME=/opt/seatunnel

ST_WEB_BASEDIR_PATH=/opt/seatunnel/ui

将connectors和plugins目录复制到web下,打包后拷贝到镜像中,然后镜像中编写的启动脚本会把该目录移动到seatunnel-zeta目录下,不然在web界面选择不了数据源,connectors又依赖plugins  

 cp -r /root/seatunnel/apache-seatunnel-2.3.3/connectors   /root/seatunnel/apache-seatunnel-web-1.0.0-bin/

 cp -r /root/seatunnel/apache-seatunnel-2.3.3/plugins   /root/seatunnel/apache-seatunnel-web-1.0.0-bin/

创建启动脚本 web自带的是后台启动,所以需要自己写个脚本在容器中前台运行

vi apache-seatunnel-web-1.0.0-bin/start.sh

一定要在bin目录下执行启动后的脚本  sh bin/seatunnel-backend-daemon.sh start ,不然找不到首页  jetty err_empty_response 404

参考:CentOs7.x安装部署SeaTunnelWeb遇到的坑_seatunnel unable to connect to any cluster-CSDN博客

<code>#!/bin/bash

cp /opt/seatunnel-zeta/lib/* /opt/seatunnel/libs/

cp /opt/seatunnel/libs/datasource*.jar /opt/seatunnel-zeta/lib/

mv -r /opt/seatunnel/connectors /opt/seatunnel-zeta/

cd /opt/seatunnel/

# 一定要在这个目录执行,不然找不到首页  jetty err_empty_response 404

# https://blog.csdn.net/qq_34905631/article/details/135074860

sh bin/seatunnel-backend-daemon.sh start

tail -f /opt/seatunnel/logs/seatunnel-web.log

创建dockerfile

FROM flink:1.18

ENV SEATUNNEL_VERSION="2.3.3"code>

ENV SEATUNNEL_WEB_VERSION="1.0.0"code>

COPY ./apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin.tar /opt/apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin.tar

WORKDIR /opt

RUN tar -xvf apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin.tar

RUN mv apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin seatunnel

RUN rm -f /opt/apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin.tar

WORKDIR /opt/seatunnel

COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz

WORKDIR /opt

RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz

RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel-zeta

RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz

RUN export SEATUNNEL_HOME=/opt/seatunnel-zeta

RUN export SEATUNNEL_WEB_HOME=/opt/seatunnel

RUN export ST_WEB_BASEDIR_PATH=/opt/seatunnel/ui

RUN export PATH=$PATH:$SEATUNNEL_HOME/bin:$SEATUNNEL_WEB_HOME/bin:$ST_WEB_BASEDIR_PATH/bin

ENTRYPOINT ["sh", "/opt/seatunnel/start.sh"]

 tar -cvf apache-seatunnel-web-1.0.0-bin.tar apache-seatunnel-web-1.0.0-bin

 docker build -t yourharbor/yourlib/seatunnel-web:1.0.0 -f dockerfile-seatunnel-web-1.0.0-bin .

 docker push  yourharbor/yourlib/seatunnel-web:1.0.0

vi web.yaml

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: seatunnel-web-deployment

  labels:

    app: seatunnel-web-dev

  namespace: seatunnel

spec:

  replicas: 1

  selector:

    matchLabels:

      app: seatunnel-web-pod

  template:

    metadata:

      labels:

        app: seatunnel-web-pod

    spec:

      containers:

      - name: seatunnel-web-serivce

        image: yourharbor/yourlib/seatunnel-web:1.0.0

        imagePullPolicy: Always

      tolerations:

      - effect: NoSchedule

        key: node-role.kubernetes.io/control-plane

---

apiVersion: v1

kind: Service

metadata:

  name: seatunnel-web-svc

  labels:

    app: seatunnel-web-svc

  namespace: seatunnel

spec:

  selector:

    app: seatunnel-web-pod

  ports:

  - name: seatunnel-web-ports

    protocol: TCP

    port: 8801

    nodePort: 3xxxx

    targetPort: 8801

  type: NodePort

部署镜像

kubectl apply -f web.yaml

错了就删除

kubectl delete -f web.yaml

查看pod

kubectl get po -n seatunnel -owide

查看svc

kubectl get svc -n seatunnel -owide

describe

kubectl describe po -n seatunnel seatunnel-web-deployment-75f465bd8b-tcvmf     

logs

kubectl logs -f -n seatunnel seatunnel-web-deployment-75f465bd8b-tcvmf      

进去容器看看

kubectl exec -it -n seatunnel seatunnel-web-deployment-75f465bd8b-tcvmf      -- /bin/bash

容器里访问一下

curl localhost:8801 

admin/admin登录,创建数据源,开始建source和sink

MySQL-CDC数据源建好了连接MySQL8.0输入连接信息时没有Driver信息,所以实现MySQL8.0 CDC会出现找不到合适的Driver的异常



声明

本文内容仅代表作者观点,或转载于其他网站,本站不以此文作为商业用途
如有涉及侵权,请联系本站进行删除
转载本站原创文章,请注明来源及作者。