目录

一、前期准备

1.1 部署 docker

1.2 部署 docker-compose

1.3 版本说明

二 、镜像构建启动

2.1 系统镜像构建

2.2 安装包源镜像构建

2.3 kdc镜像构建

2.4 集群安装

2.5 容器导出为镜像

三、Ubuntu环境安装测试

3.1 环境准备

3.2 集群容器启动

一、前期准备

1.1 部署 docker

<code># 安装yum-config-manager配置工具

yum -y install yum-utils

# 建议使用阿里云yum源：（推荐）

#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# 安装docker-ce版本

yum install -y docker-ce

# 启动并开机启动

systemctl enable --now docker

docker --version

docker info

1.2 部署 docker-compose

<code>curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

docker-compose --version

创建网络

<code>docker network create hadoop-network

1.3 版本说明

如下安装只做测试。

二 、镜像构建启动

2.1 系统镜像构建

# 拉取 Centos 镜像

FROM centos:7.6.1810

# 作者信息

LABEL kangll

# 安装基本工具

RUN yum update -y && yum install -y \

net-tools \

vim \

wget \

curl \

lrzsz \

nc \

openssl \

sed \

telnet \

openssh-server \

httpd \

chronyd \

zip unzip \

openssh-clients \

expect \

krb5-server \

krb5-libs \

krb5-workstation \

&& yum clean all

# JDK

RUN mkdir -p /usr/java

ADD ./config/jdk-8u162-linux-x64.tar.gz /usr/java/

ENV JAVA_HOME=/usr/java/jdk1.8.0_162

ENV CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

ENV PATH=$JAVA_HOME/bin:$PATH

# 添加一个名为myuser的用户

RUN useradd -m winner_spark

# 设置myuser用户的密码，这里设置为mypassword

RUN echo 'winner_spark:123456' | chpasswd

# 设置myuser用户为sudoer

RUN echo 'winner_spark ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

# SSH

RUN sed -i '/^HostKey/'d /etc/ssh/sshd_config

RUN echo 'HostKey /etc/ssh/ssh_host_rsa_key' >> /etc/ssh/sshd_config

RUN mkdir /var/run/sshd/

RUN echo 'UseDNS no' >> /etc/ssh/sshd_config

RUN sed -i -e '/pam_loginuid.so/d' /etc/pam.d/sshd

COPY ./config/hosts /etc/

#set default password

RUN echo 'root:123456' | chpasswd

RUN /usr/bin/ssh-keygen -A

COPY ./config/krb5.conf /etc

#声明22端口

EXPOSE 22

RUN mkdir -p /opt && mkdir -p /var/log/supervisord

RUN echo '/usr/sbin/sshd -D' >> /opt/run.sh

RUN chmod +x /opt/run.sh

CMD ["/opt/run.sh"]

#

WORKDIR /opt/

系统镜像构建

docker build -f dockerfile-centos -t centos-jdk:1.0 .

2.2 安装包源镜像构建

dockerfile-httpd

FROM centos-jdk:1.0

# centos7.6 JDK1.8

LABEL kangll

RUN yum -y install httpd

RUN echo "crushlinux" >/var/www/html/index.html

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

# 添加yum 源

COPY ./repo/ambari.repo /etc/yum.repos.d/

COPY ./repo/hdp.repo /etc/yum.repos.d/

COPY ./repo/hdp.gpl.repo /etc/yum.repos.d/

RUN mkdir -p /hadoop/hdp

ADD ./HDP/ambari.tar.gz /var/www/html

ADD ./HDP/HDP-GPL.tar.gz /var/www/html

ADD ./HDP/HDP.tar.gz /var/www/html

ADD ./HDP/HDP-UTILS.tar.gz /var/www/html

EXPOSE 80

CMD ["httpd","-DFOREGROUND"]

源镜像构建

docker build -t ambari:2.7.4 -f dockerfile-ambari .

2.3 kdc镜像构建

dockerfile-kerberos

<code>FROM centos:7.6.1810

RUN yum install -y krb5-server krb5-libs krb5-auth-dialog krb5-workstation

RUN yum install -y net-tools telnet

COPY ./config/krb5.conf /etc

COPY ./config/kdc.conf /var/kerberos/krb5kdc/

COPY ./config/kadm5.acl /var/kerberos/krb5kdc/

CMD ["/usr/sbin/init"]

服务启动

# 进入容器 启动服务

Systemctl restart krb5kdc

Systemctl restart kadmin

docker-compose-base.yml 启动基础环境

version: '3'

services:

mysql:

image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7

restart: always

networks:

- hadoop-network

container_name: mysql

hostname: mysql-test

environment:

MYSQL_ROOT_PASSWORD: Winner001

TZ: Asia/Shanghai

ports:

- 3306:3306

volumes:

- /data:/var/lib/mysql

- ./conf/my.cnf:/etc/mysql/my.cnf

command:

--max_connections=1000

--character-set-server=utf8mb4

--collation-server=utf8mb4_general_ci

--default-authentication-plugin=mysql_native_password

healthcheck:

test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"]

interval: 10s

timeout: 10s

httpd:

container_name: httpd

hostname: httpd

image: httpd:1.0

networks:

- hadoop-network

ports:

- "80:80"

privileged: true

restart: always

volumes:

- "/etc/hosts:/etc/hosts"

kdc:

image: kerberos:1.0

restart: always

networks:

- hadoop-network

container_name: kerberos

hostname: kerberos.winner

ports:

- 88:88

- 749:749

- 750:750

volumes:

- /etc/hosts:/etc/hosts

- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc

- /hadoop/kerberos/keytabs:/etc/security/keytabs

command:

- /bin/sh

- -c

- |

/opt/apache/kerberosStart.sh

networks:

hadoop-network:

external:

name: hadoop-network

Ambari启动

# 启动基础环境

docker-compose -f docker-compose-base.yaml up -d

# 启动ambari , 在此容器中安装集群

docker-compose -f docker-compose-ambari.yaml up -d

2.4 集群安装

repo地址

节点和秘钥

agent安装

服务安装

由于是单机直接下一步

安装组件

输入密码

数据目录确认

用户确认下一步

服务参数确定

发布

安装完成

2.5 容器导出为像

<code>

docker ps -a

docker commit 容器ID ambari-1.0.tar.gz

三、Ubuntu环境安装测试

3.1 环境准备

我们准备Ubuntu 环境，在Ubuntu环境安装Ambari单机，相当于在Centos服务器安装的docker化的Ambari 导出为镜像 然后在Ubutntu 环境尝试启动。 

准备Ubuntu 环境，版本：Ubuntu 20.04.5 LTS

将镜像 ambari-1.0.tar.gz 加载到Ubuntu 环境

<code>docker load -i ambari-1.0.tar.gz

MySQL元数据库备份还原（Ambari、hive、ranger）

HDFS、kerberos 备份的数据拷贝到Ubuntu服务器并保持数据文件路径不变。

将在centos 安装完成的集群备份元数据库写入到新的库中 , 也就是Ubuntu环境要使用的元数据库。

# 备份的元数据写入新的数据库中

mysql -hmysql-test -uroot -pWinner001 ambari < ambari.sql

mysql -hmysql-test -uroot -pWinner001 ranger < ranger.sql

mysql -hmysql-test -uroot -pWinner001 hive < hive.sql

备份的kerberos和安装包解压

# 解压到 /usr 路径下，保存的是集群安装的数据

hdp.tar.gz

# 在根路径解压保留的kerberos 和HDFS 初始化的元数据

hadoop.tar.gz

基础环境启动docker-compose-base.yaml

<code>version: '3'

services:

mysql:

image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7

restart: always

networks:

- hadoop-network

container_name: mysql

hostname: mysql-test

environment:

MYSQL_ROOT_PASSWORD: Winner001

TZ: Asia/Shanghai

ports:

- 3306:3306

volumes:

- /data:/var/lib/mysql

- ./conf/my.cnf:/etc/mysql/my.cnf

command:

--max_connections=1000

--character-set-server=utf8mb4

--collation-server=utf8mb4_general_ci

--default-authentication-plugin=mysql_native_password

healthcheck:

test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"]

interval: 10s

timeout: 10s

httpd:

container_name: httpd

hostname: httpd

privileged: true

image: httpd:1.0

networks:

- hadoop-network

ports:

- "80:80"

privileged: true

restart: always

volumes:

- "/etc/hosts:/etc/hosts"

kdc:

image: kerberos:1.0

restart: always

privileged: true

networks:

- hadoop-network

container_name: kerberos

hostname: kerberos.winner

ports:

- 888:88

- 8749:749

- 8750:750

volumes:

- /etc/hosts:/etc/hosts

- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc

- /hadoop/kerberos/keytabs:/etc/security/keytabs

networks:

hadoop-network:

external:

name: hadoop-network

启动

# 启动

docker-compose -f docker-compose-base.yaml up -d

# 停止

docker-compose -f docker-compose-base.yaml down

3.2 集群容器启动

镜像

启动容器

<code>version: '3'

services:

ambari:

image: ambari:2.7.4 # 修改为新的容器

container_name: ambari

hostname: ambari-server

restart: always

privileged: true

networks:

- hadoop-network

ports:

- 8080:8080

- 8088:8088

- 50070:50070

- 50075:50075

- 8188:8188

- 8042:8042

- 10000:10000

- 9083:9083

- 6080:6080

- 16010:16010

volumes:

- /etc/hosts:/etc/hosts

- /usr/hdp:/usr/hdp

- /hadoop/hdfs:/hadoop/hdfs

- /hadoop/etc/hadoop/:/etc/hadoop/

- /hadoop/etc/hbase/:/etc/hbase/

- /hadoop/etc/hive/:/etc/hive/

- /hadoop/etc/livy2/:/etc/livy2/

- /hadoop/etc/spark2/:/etc/spark2/

- /hadoop/etc/tez/:/etc/tez/

- /hadoop/etc/zookeeper/:/etc/zookeeper/

- /hadoop/yarn:/hadoop/yarn

- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc

- /hadoop/kerberos/keytabs:/etc/security/keytabs

command:

- /bin/sh

- -c

- |

/opt/apache/bootstrap.sh

networks:

hadoop-network:

external:

name: hadoop-network

启动

docker-compose -f docker-compose-ambari.yaml up -d

下次安装直接加载镜像，然后执行yaml文件即可

<code>docker load -i ambari-1.0.tar.gz

docker load -i kerberos.tar.gz

docker load -i mysql.tar.gz

docker load -i HDP-REPO.tar.gz

脚本

bootstrap.sh

#!/bin/bash

set -x

ambari_db=ambari

myurl=mysql-test

myuser=root

mypwd=Winner001

myport=3306

mydb=ipvacloud

java_home=/usr/java/jdk1.8.0_162

mysql_driver=/usr/share/java/mysql-connector-java.jar

BASEDIR=$(cd "$(dirname "$0")"; pwd)

wait_for() {

echo Waiting for $1 to listen on $2...

while ! nc -z $1 $2; do echo waiting...; sleep 1s; done

}

setup_ambari() {

# 强制替换文件

cp -f /tmp/ambari.properties /etc/ambari-server/conf/

cp -f /tmp/password.dat /etc/ambari-server/conf/

cp -f /tmp/ambari-agent.ini /etc/ambari-agent/conf/

}

# wait for mysql and httpd

#wait_for windp-aio 3306

#wait_for httpd 80

# 开始ambari安装

yum -y install ambari-server

yum -y install ambari-agent

re=`mysql -h${myurl} -u${myuser} -p${mypwd} -e "show databases"|grep ambari|wc -l`

if [ $re -eq 0 ];then

# sql 初始化

mysql -h${myurl} -u${myuser} -p${mypwd} < /tmp/init_db.sql

mysql -h${myurl} -u${myuser} -p${mypwd} ambari < /tmp/Ambari-DDL-MySQL-CREATE.sql

fi

# 配置

setup_ambari

# 启动ambari服务

/usr/sbin/ambari-server start

/usr/sbin/ambari-agent start

/usr/sbin/sshd -D

tail -f /var/log/ambari-server/ambari-server.log

参考文章：https://www.cnblogs.com/liugp/p/17489616.html

Ubuntu 环境安装docker: 【云计算】Ubuntu多种安装docker方式_ubuntu安装docker-CSDN博客