flink + iceberg 快速搭建

the environment includes:

• minio
• iceberg
• flink

Centos 更换 tencent 的yum源

备份系统旧配置文件

<code>mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

获取对应版本的CentOS-Base.repo 到/etc/yum.repos.d/目录

各版本源配置列表

CentOS7

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos7_base.repo

CentOS8

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos8_base.repo

更新缓存

yum clean all

yum makecache

Centos 安装Java环境（flink使用）

wget --no-check-certificate https://repo.huaweicloud.com/java/jdk/8u202-b08/jdk-8u202-linux-x64.tar.gz

mkdir /usr/local/java/

tar -zxvf jdk-8u202-linux-x64.tar.gz -C /usr/local/java

echo "export JAVA_HOME=/usr/local/java/jdk1.8.0_202" >> /etc/profile

echo "export JRE_HOME=${JAVA_HOME}/jre" >> /etc/profile

echo "export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib" >> /etc/profile

echo "export PATH=${JAVA_HOME}/bin:$PATH" >> /etc/profile

source /etc/profile

ln -s /usr/local/java/jdk1.8.0_202/bin/java /usr/bin/java

java -version

Centos install Docker & Docker Compose

First, To install Docker Engine, you need a maintained version of CentOS 7 or 8.

Archived versions aren't supported or tested.

sudo yum install -y python3-pip yum-utils device-mapper-persistent-data lvm2

sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

sudo yum install docker-ce

sudo systemctl start docker

update docker daemon..

sudo mkdir -p /etc/docker

sudo tee /etc/docker/daemon.json <<-'EOF'

{

"registry-mirrors": [

"https://docker.fxxk.dedyn.io/",

"https://dockerproxy.cn"

]

}

EOF

sudo systemctl daemon-reload

sudo systemctl restart docker

install docker compose..

sudo curl -L "https://mirrors.aliyun.com/docker-toolbox/linux/compose/1.21.2/docker-compose-Linux-x86_64" -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

docker-compose --version

install Minio

使用以下命令下载安装最新版本的稳定 MinIO二进制包， 并设置$PATH:

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio

chmod +x minio

sudo mv minio /usr/local/bin/

设置minio linux服务

sudo tee /etc/systemd/system/minio.service <<-'EOF'

[Unit]

Description=Minio Service

[Service]

Environment="MINIO_ROOT_USER=admin"code>

Environment="MINIO_ROOT_PASSWORD=password"code>

ExecStart=/usr/local/bin/minio server /mnt/minio --console-address ":9001"

ExecReload=/bin/kill -s HUP $MAINPID

ExecStop=/bin/kill -s QUIT $MAINPID

StandardOutput=/mnt/minio/logs/minio.log

PrivateTmp=true

[Install]

WantedBy=multi-user.target

EOF

启动 minio linux服务

systemctl start minio

systemctl enable minio

install iceberg in docker

save the yaml below into a file nameddocker-compose.yml:

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio

version: "3"

services:

rest:

image: tabulario/iceberg-rest:1.5.0

container_name: iceberg-rest

networks:

iceberg_net:

ports:

- 8181:8181

environment:

- AWS_ACCESS_KEY_ID=admin

- AWS_SECRET_ACCESS_KEY=password

- AWS_REGION=us-east-1

- CATALOG_WAREHOUSE=s3://warehouse/

- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO

- CATALOG_S3_ENDPOINT=<更换本地minio服务地址:9000>

mc:

image: minio/mc

container_name: mc

networks:

iceberg_net:

environment:

- AWS_ACCESS_KEY_ID=admin

- AWS_SECRET_ACCESS_KEY=password

- AWS_REGION=us-east-1

entrypoint: >

/bin/sh -c "

until (/usr/bin/mc config host add minio http://<更换本地minio服务地址:9000> admin password) do echo '...waiting...' && sleep 1; done;

/usr/bin/mc mb minio/warehouse;

/usr/bin/mc policy set public minio/warehouse;

tail -f /dev/null

"

networks:

iceberg_net:

Next, start up the docker containers with this command:

docker-compose up -d

install flink

download jdk filnk 1.18.1

wget https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.18.1/flink-1.18.1-bin-scala_2.12.tgz

tar zxvf flink-1.18.1-bin-scala_2.12.tgz -C /opt/

download dll ，copy to flink-1.18.1/lib folder

wget https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.5.2/iceberg-aws-bundle-1.5.2.jar

wget ttps://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.18/1.5.2/iceberg-flink-runtime-1.18-1.5.2.jar

wget https://repo1.maven.org/maven2/org/apache/flink/flink-connector-jdbc/3.1.2-1.18/flink-connector-jdbc-3.1.2-1.18.jar

wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar

wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.18.0/flink-s3-fs-hadoop-1.18.0.jar

cp ./*.jar /opt/flink-1.18.1/lib/

start flink

cd /opt/flink-1.18.1/

bash ./bin/start-cluster.sh

flink 创建icberg表

bash flink 客户端：

cd /opt/flink-1.18.1/

bash ./bin/sql-client.sh

SET 'sql-client.execution.result-mode' = 'tableau';

SET 'execution.runtime-mode' = 'batch';

创建 Iceberg Catalog

CREATE CATALOG iceberg WITH (

'type'='iceberg',

'catalog-type'='rest',

'uri'='http://rest:8181/',

's3.endpoint'='http://<更换本地minio服务地址:9000>',

'warehouse'='s3://warehouse/wh/'

);

创建数据库和 Iceberg 表

create database if not exists iceberg.db_iceberg;

CREATE TABLE if not exists iceberg.db_iceberg.tb_iceberg (

id BIGINT,

val string,

PRIMARY KEY (id) NOT ENFORCED

) WITH (

'write.upsert.enabled'='true',

'upsert-enabled'='true',

'write.delete.mode'='merge-on-read',

'write.update.mode'='merge-on-read'

);