SeaTunnel-web in K8S
crabdave123 2024-07-27 13:33:02 阅读 68
下载,官网下载有问题,上dlcdn.apache.org下载
https://dlcdn.apache.org/seatunnel/seatunnel-web/1.0.0/apache-seatunnel-web-1.0.0-bin.tar.gz
apache-seatunnel-2.3.3中执行bin/install-plugin.sh下载connectors
下载web的源码
https://github.com/apache/seatunnel-web.git
seatunnel-server/seatunnel-app/src/main/bin/download_datasource.sh
将download_datasource.sh复制到/root/seatunnel/apache-seatunnel-web-1.0.0-bin/bin目录
sh download_datasource.sh 下载 datasource
vim conf/application.yml修改端口号和数据源信息
根据script目录下的数据表初始化脚本,把数据库建好 seatunnel_server_mysql.sql
将apache-seatunnel-2.3.4/conf目录下的hazelcast-client.yaml和plugin-mapping.properties复制过来
hazelcast-client.yaml中的信息连接到之前部署好的SeaTunnel集群
参考:SeaTunnel 2.3.4 Cluster in K8S-CSDN博客
logback-spring.xml文件中的日志级别如果需要可改成debug方便调试
准备创建镜像文件
apache-seatunnel-2.3.3(重命名为:seatunnel-zeta)和apache-seatunnel-web-1.0.0-bin(重命名为:seatunnel,web依赖zeta连接集群)都放在一个容器中。
在镜像中配置相关的环境变量
SEATUNNEL_HOME=/opt/seatunnel-zeta
SEATUNNEL_WEB_HOME=/opt/seatunnel
ST_WEB_BASEDIR_PATH=/opt/seatunnel/ui
将connectors和plugins目录复制到web下,打包后拷贝到镜像中,然后镜像中编写的启动脚本会把该目录移动到seatunnel-zeta目录下,不然在web界面选择不了数据源,connectors又依赖plugins
cp -r /root/seatunnel/apache-seatunnel-2.3.3/connectors /root/seatunnel/apache-seatunnel-web-1.0.0-bin/
cp -r /root/seatunnel/apache-seatunnel-2.3.3/plugins /root/seatunnel/apache-seatunnel-web-1.0.0-bin/
创建启动脚本 web自带的是后台启动,所以需要自己写个脚本在容器中前台运行
vi apache-seatunnel-web-1.0.0-bin/start.sh
一定要在bin目录下执行启动后的脚本 sh bin/seatunnel-backend-daemon.sh start ,不然找不到首页 jetty err_empty_response 404
参考:CentOs7.x安装部署SeaTunnelWeb遇到的坑_seatunnel unable to connect to any cluster-CSDN博客
<code>#!/bin/bash
cp /opt/seatunnel-zeta/lib/* /opt/seatunnel/libs/
cp /opt/seatunnel/libs/datasource*.jar /opt/seatunnel-zeta/lib/
mv -r /opt/seatunnel/connectors /opt/seatunnel-zeta/
cd /opt/seatunnel/
# 一定要在这个目录执行,不然找不到首页 jetty err_empty_response 404
# https://blog.csdn.net/qq_34905631/article/details/135074860
sh bin/seatunnel-backend-daemon.sh start
tail -f /opt/seatunnel/logs/seatunnel-web.log
创建dockerfile
FROM flink:1.18
ENV SEATUNNEL_VERSION="2.3.3"code>
ENV SEATUNNEL_WEB_VERSION="1.0.0"code>
COPY ./apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin.tar /opt/apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin.tar
WORKDIR /opt
RUN tar -xvf apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin.tar
RUN mv apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin seatunnel
RUN rm -f /opt/apache-seatunnel-web-${SEATUNNEL_WEB_VERSION}-bin.tar
WORKDIR /opt/seatunnel
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel-zeta
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN export SEATUNNEL_HOME=/opt/seatunnel-zeta
RUN export SEATUNNEL_WEB_HOME=/opt/seatunnel
RUN export ST_WEB_BASEDIR_PATH=/opt/seatunnel/ui
RUN export PATH=$PATH:$SEATUNNEL_HOME/bin:$SEATUNNEL_WEB_HOME/bin:$ST_WEB_BASEDIR_PATH/bin
ENTRYPOINT ["sh", "/opt/seatunnel/start.sh"]
tar -cvf apache-seatunnel-web-1.0.0-bin.tar apache-seatunnel-web-1.0.0-bin
docker build -t yourharbor/yourlib/seatunnel-web:1.0.0 -f dockerfile-seatunnel-web-1.0.0-bin .
docker push yourharbor/yourlib/seatunnel-web:1.0.0
vi web.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: seatunnel-web-deployment
labels:
app: seatunnel-web-dev
namespace: seatunnel
spec:
replicas: 1
selector:
matchLabels:
app: seatunnel-web-pod
template:
metadata:
labels:
app: seatunnel-web-pod
spec:
containers:
- name: seatunnel-web-serivce
image: yourharbor/yourlib/seatunnel-web:1.0.0
imagePullPolicy: Always
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
---
apiVersion: v1
kind: Service
metadata:
name: seatunnel-web-svc
labels:
app: seatunnel-web-svc
namespace: seatunnel
spec:
selector:
app: seatunnel-web-pod
ports:
- name: seatunnel-web-ports
protocol: TCP
port: 8801
nodePort: 3xxxx
targetPort: 8801
type: NodePort
部署镜像
kubectl apply -f web.yaml
错了就删除
kubectl delete -f web.yaml
查看pod
kubectl get po -n seatunnel -owide
查看svc
kubectl get svc -n seatunnel -owide
describe
kubectl describe po -n seatunnel seatunnel-web-deployment-75f465bd8b-tcvmf
logs
kubectl logs -f -n seatunnel seatunnel-web-deployment-75f465bd8b-tcvmf
进去容器看看
kubectl exec -it -n seatunnel seatunnel-web-deployment-75f465bd8b-tcvmf -- /bin/bash
容器里访问一下
curl localhost:8801
admin/admin登录,创建数据源,开始建source和sink
MySQL-CDC数据源建好了连接MySQL8.0输入连接信息时没有Driver信息,所以实现MySQL8.0 CDC会出现找不到合适的Driver的异常
上一篇: C# MES通信从入门到精通(8)——C#调用Webservice服务进行数据交互
下一篇: GitLab16.8配置webhooks、Jenkins2.4配置GitLab插件实现持续集成、配置宝塔面板实现持续部署(其三)
本文标签
声明
本文内容仅代表作者观点,或转载于其他网站,本站不以此文作为商业用途
如有涉及侵权,请联系本站进行删除
转载本站原创文章,请注明来源及作者。