【k8s】Kubernetes 1.29.4离线安装部署（总）

轻云UC 2024-06-20 17:37:02 阅读 98

（一）kubernetes1.29.4离线部署之-安装文件准备

（二）kubernetes1.29.4离线部署之-镜像文件准备

（三）kubernetes1.29.4离线部署之-环境初始化

（四）kubernetes1.29.4离线部署之-组件安装

（五）kubernetes1.29.4离线部署之-初始化第一个控制平面

（六）kubernetes1.29.4离线部署之-加入Node节点

（七）kubernetes1.29.4离线部署之-网络插件

（八）kubernetes1.29.4离线部署之-测试验证

友情提示：由于本文过长，不便于阅读，已经拆分为多个章节，可以点击上面的地址单独阅读

本文容器运行时采用的时Containerd

本文网络插件采用的时calico tigera-operator

本文涉及的所有脚本文件可以从如下地址获取：

https://gitee.com/qingplus/qingcloud-platform/tree/develop/qingcloud-deploy/service

前情提要：

本文不介绍K8S架构及细节，仅仅分享Kubernetes部署实施完整过程。

本文主要采用kubeadm方式安装部署。另外本文直接采用单控制平面方式快速完成版本部署，etcd的部署方式为堆叠方式，独立方式的切换后续完成文档说明。

高可用集群拓扑的两个选项介绍：

高可用集群拓扑的两个选项：

使用堆叠（stacked）控制平面节点，其中 etcd 节点与控制平面节点共存使用外部 etcd 节点，其中 etcd 在与控制平面不同的节点上运行

本文主要采用第一种堆叠方式：

环境准备

硬件环境

服务器由6台ESXi虚拟化服务器组成，详情参考《ESXi网络配置与物理交换机VLAN ID规划》

本文为单控制平面部署，非高可用集群部署，并未用到规划中的所有节点。后续集群部署会继续使用。

主机名称	ESXi 节点名	角色	IP地址	资源配置
itserver-master1	xenserver01	master	10.0.0.10	8C/16G/200G
itserver-master2	xenserver02	master	10.0.0.13	8C/16G/200G
itserver-node01	xenserver01	node	10.0.0.11	16C/64G/500G
itserver-node02	xenserver01	node	10.0.0.12	16C/64G/500G
itserver-node03	xenserver02	node	10.0.0.14	16C/64G/500G
itserver-node04	xenserver02	node	10.0.0.15	16C/64G/500G

物理服务器节点（安装ESXi）

三台物理服务器
ESXi 节点名	ip地址	用户名	备注
xenserver01	192.168.3.50	root
xenserver02	192.168.3.60	root
xenserver03	192.168.3.100	root

网络分配

角色	IP地址	备注
node网络	10.0.0.0/24
Service 网络	10.96.0.0/16
Pod网络	172.16.0.0/16

网络部署图

企业基础网络结构拓扑.png

离线安装文件准备

需要准备如下几个必须的安装文件，可以离线下载后通过本地nginx代理供其他节点下载使用

kubernetes_server

直接下载最新版二进制文件

containerd

本文容器运行时采用containerd而非docker，如果容器运行时采用docker，需要另外的部署版本。本文直接下载三合一版本cri_containerd_cni

cni_pluginscrictl

命令行执行工具。类似docker命令行

runc

需要单独下载完整版，只是需要检查本地环境是否存在libcommp.so

etcd

采用外部 etcd 节点部署时需要用到，如果使用的是堆叠的方式，直接用默认的etcd容器即可，可以不用下载。

完整的离线文件下载脚本：

#!/bin/bashENV_CFG=./env.cfgif [ -f ${ENV_CFG} ] ; thenchmod 777 ${ENV_CFG}source ${ENV_CFG}fi# Internet URLskernel_url="http://mirrors.tuna.tsinghua.edu.cn/elrepo/kernel/el7/x86_64/RPMS/${kernel_name}"cni_plugins_url="https://github.com/containernetworking/plugins/releases/download/${cni_plugins_version}/${cni_plugins_name}"cri_containerd_cni_url="https://github.com/containerd/containerd/releases/download/v${cri_containerd_cni_version}/${cri_containerd_cni_name}"crictl_url="https://github.com/kubernetes-sigs/cri-tools/releases/download/${crictl_version}/${crictl_name}"runc_url="https://github.com/opencontainers/runc/releases/download/v${runc_version}/${runc_name}"etcd_url="https://github.com/etcd-io/etcd/releases/download/${etcd_version}/${etcd_name}"kubernetes_server_url="https://storage.googleapis.com/kubernetes-release/release/${KUBERNETES_VERSION}/${kubernetes_server_name}"nginx_url="http://nginx.org/download/${nginx_name}"# Download packagespackages=( $kernel_url $runc_url $cni_plugins_url $cri_containerd_cni_url $crictl_url $cri_dockerd_url $etcd_url $kubernetes_server_url)for package_url in "${packages[@]}"; do filename=$(basename "$package_url") if curl -k -L -C - -o "$filename" "$package_url"; then echo "Downloaded $filename" else echo "Failed to download $filename" exit 1 fidone

离线镜像文件准备

注意：离线安装需要准备大量的镜像，这一步千万要仔细否则会出现各种意想不到的问题

需要准备的镜像文件

kube-apiserverkube-controller-managerkube-schedulerkube-proxykube-proxycorednspauseetcd

以上镜像文件可以从： registry.cn-hangzhou.aliyuncs.com/google_containers下载

calico/nodecalico/kube-controllerscalico/nodecalico/typhacalico/node-driver-registrarcalico/csicalico/cnicalico/ctlcalico/pod2daemon-flexvolcalico/apiserver

以上可以直接从docker.io下载即可

注意：以上所有的镜像文件务必准备到位。本文脚本中的几个版本便令注意修改到位：

KUBERNETES_VERSION=${KUBERNETES_VERSION:-“v1.29.4”}

COREDNS_VERSION=${COREDNS_VERSION:-‘v1.11.1’}

PAUSE_VERSION=${PAUSE_VERSION:-‘3.9’}

ETCD_VERSION=${ETCD_VERSION:-‘3.5.12-0’}

REGISTRY_VERSION=${REGISTRY_VERSION:-‘2.8.3’}

CALICO_VERSION=${CALICO_VERSION:-‘v3.27.3’}

完整的镜像下载脚本

#!/bin/bashENV_CFG=./env.cfgif [ -f ${ENV_CFG} ] ; thenchmod 777 ${ENV_CFG}source ${ENV_CFG}fiimage_list="${IMAGE_DOMAIN}/${IMAGE_NAMESPACE}/kube-apiserver:${KUBERNETES_VERSION}${IMAGE_DOMAIN}/${IMAGE_NAMESPACE}/kube-controller-manager:${KUBERNETES_VERSION}${IMAGE_DOMAIN}/${IMAGE_NAMESPACE}/kube-scheduler:${KUBERNETES_VERSION}${IMAGE_DOMAIN}/${IMAGE_NAMESPACE}/kube-proxy:${KUBERNETES_VERSION}${IMAGE_DOMAIN}/${IMAGE_NAMESPACE}/coredns:${COREDNS_VERSION}${IMAGE_DOMAIN}/${IMAGE_NAMESPACE}/pause:${PAUSE_VERSION}${IMAGE_DOMAIN}/${IMAGE_NAMESPACE}/etcd:${ETCD_VERSION}calico/node:${CALICO_VERSION}calico/kube-controllers:${CALICO_VERSION}calico/node:${CALICO_VERSION}calico/typha:${CALICO_VERSION}calico/node-driver-registrar:${CALICO_VERSION}calico/csi:${CALICO_VERSION}calico/cni:${CALICO_VERSION}calico/ctl:${CALICO_VERSION}calico/pod2daemon-flexvol:${CALICO_VERSION}calico/apiserver:${CALICO_VERSION}"#${IMAGE_DOMAIN}/${IMAGE_NAMESPACE}/registry:${REGISTRY_VERSION}newimage_list=()for image in ${image_list}; do docker pull "${image}" newimage=$(echo $image | sed -e "s/calico/${LOCAL_IMAGE_DOMAIN}\/calico/") newimage=$(echo $newimage | sed -e "s/${IMAGE_DOMAIN}\/${IMAGE_NAMESPACE}/${LOCAL_IMAGE_DOMAIN}\/${LOCAL_IMAGE_NAMESPACE}/") newimage_list+="${newimage} " docker tag $image $newimage docker push $newimagedonedocker save -o qinghub-kube-"${VERSION}".tar ${newimage_list}

轻云官方下载

https:/qingplus.cn/pkg/kubernetes/v1.29.4/qinghub-kube-v1.29.4.tar

下载完成后再手动导入所有的镜像即可。详细过程不在细说。

环境初始化

检查步骤

关闭防火墙关闭 swap partition permanently配置检查时间同步配置安装时间同步组件配置检查 nfs-utils kubeadmin方式安装不需要检查配置检查内核版本配置检查资源情况配置检查SSH配置检查系统配置配置检查转发 IPv4配置检查Docker用户并添加ssh免密认证<authoirzed_keys> （建议手动执行）配置检查Docker （容器运行时为Containerd时，不需要检查）配置检查Docker用户权限（容器运行时为Containerd时，不需要检查）配置检查网络

完整的初始化脚本

#!/bin/bash################################################ QingHub K8S Install 版本: $VERSION# 架构: $ARCH_TYPE 目前版本主要支持amd64，其他待敬请期待# 操作系统: $os_type# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################ENV_CFG=./env.cfgif [ -f ${ENV_CFG} ] ; thenchmod 777 ${ENV_CFG}source ${ENV_CFG}fiexport CONSOLE=${CONSOLE:-false}os_type=$(cat /etc/os-release | grep "^ID=" | awk -F= '{print $2}' | tr -d [:punct:])os_version_id=$(cat /etc/os-release | grep "VERSION_ID=" | awk -F= '{print $2}' | tr -d [:punct:])if [ "$EUID" -ne 0 ]; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${RED}[ERROR] 当前用户不是 root 用户，请切换到 root 用户执行该脚本.${NC}" exit 1 else echo -e "${RED}[ERROR] Current user is not root user, please switch to root user to execute the script.${NC}" exit 1 fifiif [ -z "$SSH_RSA" ]; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${RED}[ERROR] 请设置环境变量 SSH_RSA, 该变量为 SSH 公钥.${NC}" exit 1 else echo -e "${RED}[ERROR] Please set the environment variable SSH_RSA, the variable is SSH public key.${NC}" exit 1 fifi################################################ 新增ubuntu 用户# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function add_user_in_ubuntu() { useradd --create-home -s /bin/bash -g root "$1" echo "$1":"$2" | chpasswd if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] 用户 $1 已经创建.${NC}" else echo -e "${GREEN}[INFO] User $1 has been created.${NC}" fi}################################################ 新增redhat 用户# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function add_user_in_redhat() { adduser -g root "$1" echo "$1":"$2" | chpasswd if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] 用户 $1 已经创建.${NC}" else echo -e "${GREEN}[INFO] User $1 has been created.${NC}" fi}################################################ 描述：检查并新增用户, 有些版本可以不用检查，请使用时根据# 情况自行注释掉# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function check_user() { if ! grep -q docker /etc/group; then groupadd --force docker fi if id -u "${DOCKER_USER}" >/dev/null 2>&1; then if ! id -nG "${DOCKER_USER}" | grep -qw "docker"; then gpasswd -a "${DOCKER_USER}" docker fi if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] 用户 ${DOCKER_USER} 已经存在.${NC}" else echo -e "${GREEN}[INFO] User ${DOCKER_USER} already exists.${NC}" fi else case $os_type in centos|redhat|euleros|fusionos|anolis|kylin|rhel|rocky|fedora|openEuler) add_user_in_redhat "${DOCKER_USER}" "${DOCKER_PASS}" ;; ubuntu|debian) add_user_in_ubuntu "${DOCKER_USER}" "${DOCKER_PASS}" ;; *) if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${RED}[ERROR] 暂不支持 $os_type 操作系统.${NC}" exit 1 else echo -e "${RED}[ERROR] The $os_type operating system is temporarily not supported.${NC}" exit 1 fi ;; esac fi $CONSOLE $CONSOLE || add_ssh_rsa "${DOCKER_USER}"}function add_ssh_rsa() { if id -u "$user" >/dev/null 2>&1; then if [ ! -d "/home/$1/.ssh" ]; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] 创建 /home/$1/.ssh 目录.${NC}" else echo -e "${GREEN}[INFO] Create /home/$1/.ssh directory.${NC}" fi mkdir -p /home/"$1"/.ssh fi if [ -f "/home/$1/.ssh/authorized_keys" ]; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] /home/$1/.ssh/authorized_keys 已经存在.${NC}" else echo -e "${GREEN}[INFO] /home/$1/.ssh/authorized_keys already exists.${NC}" fi chmod 777 /home/"$1"/.ssh/authorized_keys if ! < /home/"$1"/.ssh/authorized_keys grep -q "$SSH_RSA"; then echo "$SSH_RSA" >> /home/"$1"/.ssh/authorized_keys fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] 创建 /home/$1/.ssh/authorized_keys.${NC}" else echo -e "${GREEN}[INFO] Create /home/$1/.ssh/authorized_keys.${NC}" fi touch /home/"$1"/.ssh/authorized_keys chmod 777 /home/"$1"/.ssh/authorized_keys echo "$SSH_RSA" > /home/"$1"/.ssh/authorized_keys fi if < /home/"$1"/.ssh/authorized_keys grep -q "$SSH_RSA"; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] 成功将 SSH 公钥添加到 /home/$1/.ssh/authorized_keys.${NC}" else echo -e "${GREEN}[INFO] Successfully added ssh public key to /home/$1/.ssh/authorized_keys.${NC}" fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${RED}[ERROR] 将 SSH 公钥添加到 /home/$1/.ssh/authorized_keys 失败.${NC}" exit 1 else echo -e "${RED}[ERROR] Add ssh public key to /home/$1/.ssh/authorized_keys failed.${NC}" exit 1 fi fi chmod 600 /home/"$1"/.ssh/authorized_keys chown -R "$1":"$1" /home/"$1"/.ssh fi}function check_user_permission(){ if su ${DOCKER_USER} -c "docker ps" >/dev/null 2>&1; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] Docker 用户有权限执行 docker 命令.${NC}" else echo -e "${GREEN}[INFO] Docker users have the permission to execute docker commands.${NC}" fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${RED}[ERROR] Docker 用户无权限执行 docker 命令, 请尝试重启docker 'systemctl restart docker'. 重启 docker 后, 再次执行该脚本.${NC}" exit 1 else echo -e "${RED}[ERROR] Docker users have no permission to execute docker commands, Please try to restart docker 'systemctl restart docker'. After restarting docker, execute the script again.${NC}" exit 1 fi fi}################################################ 描述：关闭防火墙# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function disable_firewalld() { if systemctl status firewalld | grep Active | grep -q running >/dev/null 2>&1; then systemctl stop firewalld >/dev/null 2>&1 systemctl disable firewalld >/dev/null 2>&1 if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] 检测到 Firewalld 服务已启动，正在将 Firewalld 服务关闭并禁用.${NC}" else echo -e "${GREEN}[INFO] The Firewalld service has been started, Firewalld service is being turned off and disabled.${NC}" fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] Firewalld 服务已经停止或未安装.${NC}" else echo -e "${GREEN}[INFO] Firewalld service is not installed.${NC}" fi fi}################################################ 描述：关闭swap# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function disable_swap() { if swapoff -a; then sed -i '/swap/s/^/#/' /etc/fstab if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] swap 已经禁用.${NC}" else echo -e "${GREEN}[INFO] swap has been disabled.${NC}" fi fi}function check_time_sync() { if timedatectl status | grep "NTP synchronized" | grep -q "yes" >/dev/null 2>&1 || timedatectl show | grep "NTPSynchronized=yes" >/dev/null 2>&1; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] NTP 时间同步已经启用.${NC}" else echo -e "${GREEN}[INFO] NTP time synchronization has been enabled.${NC}" fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] NTP 时间同步未启用.${NC}" else echo -e "${YELLOW}[WARN] NTP time synchronization is not enabled.${NC}" fi fi}################################################ 描述：安装时钟同步，请酌情修改并安装# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################install_chrony(){ case $os_type in ubuntu|debian) if dpkg -l | grep -q chrony >/dev/null 2>&1; then echo -e "${GREEN}[INFO] chrony 已经安装在主机上.${NC}" else echo -e "${YELLOW}[WARN] chrony 未安装在主机上, 请执行命令安装 'apt -y install chrony'.${NC}" apt -y install chrony &> /dev/null; systemctl restart chronyd && systemctl enable --now chronyd &> /dev/null systemctl is-active chronyd &> /dev/null fi ;; *) if rpm -qa | grep -q chrony >/dev/null 2>&1; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] chrony 已经安装在主机上.${NC}" else echo -e "${GREEN}[INFO] chrony has been installed on the host.${NC}" fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] chrony 未安装在主机上, 请执行命令安装 'yum -y install chrony'.${NC}" else echo -e "${YELLOW}[WARN] chrony is not installed on the host, please execute the command install 'yum -y install chrony'.${NC}" fi yum -y install chrony fi ;; esac if [ "${CHRONY_TYPE}" == 'server' ]; then sudo bash -c 'cat > /etc/chrony.conf << EOFpool ntp.aliyun.com iburstdriftfile /var/lib/chrony/driftmakestep 1.0 3rtcsyncallow 10.0.0.0/24local stratum 10keyfile /etc/chrony.keysleapsectz right/UTClogdir /var/log/chronyEOF' else sudo bash -c 'cat > /etc/chrony.conf << EOFpool ${CHRONY_SERVER} iburstdriftfile /var/lib/chrony/driftmakestep 1.0 3rtcsynckeyfile /etc/chrony.keysleapsectz right/UTClogdir /var/log/chronyEOF' fi systemctl restart chronyd && systemctl enable --now chronyd &> /dev/null systemctl is-active chronyd &> /dev/null if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] chrony 完成配置在主机上.${NC}" else echo -e "${GREEN}[INFO] chrony has been configured on the host.${NC}" fi}################################################ 描述：优化配置forwarding_ipv4# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function check_forwarding_ipv4() { sudo bash -c 'cat <<EOF | sudo tee /etc/modules-load.d/k8s.confoverlaybr_netfilterEOF' sudo modprobe overlay sudo modprobe br_netfilter sudo bash -c 'cat <<EOF | sudo tee /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-iptables = 1net.bridge.bridge-nf-call-ip6tables = 1net.ipv4.ip_forward = 1EOF' sudo sysctl --system #加载ipvs模块 sudo bash -c 'cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf <<EOFip_vsip_vs_rrip_vs_wrrip_vs_shnf_conntrackip_tablesip_setxt_setipt_setipt_rpfilteript_REJECTipipEOF' systemctl restart systemd-modules-load.service}################################################ 描述：检查服务器资源状况# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function check_resource(){ cpu=$(grep -c 'processor' /proc/cpuinfo) mem=$(free -g | awk '/^Mem/{print $2}') DISK_SPACE=$(df /|sed -n '2p'|awk '{print $2}') # check cpu if [ "${cpu}" -lt 2 ]; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] CPU核数建议至少为2核.${NC}" else echo -e "${YELLOW}[WARN] The cpu is recommended to be at least 2C.${NC}" fi fi # check memory if [ "${mem}" -lt 3 ]; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] 内存建议至少为8G.${NC}" else echo -e "${YELLOW}[WARN] The Memory is recommended to be at least 8G.${NC}" fi fi # check disk space if [ "${DISK_SPACE}" -lt 47185920 ];then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] 根分区空间需大于 50G.${NC}" else echo -e "${YELLOW}[WARN] The root partition space must be greater than 50G.${NC}" fi fi}################################################ 描述：检查内核版本# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function check_kernel() { kernel_version=$(uname -r | awk -F. '{print $1}') if [ "$kernel_version" -lt "4" ]; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] 内核版本必须高于4.0, 请尽快升级内核到4.0+.${NC}" else echo -e "${YELLOW}[WARN] Kernel version must be higher than 4.0, Please upgrade the kernel to 4.0+ as soon as possible.${NC}" fi fi}################################################ 描述：检查 nfs是否安装，这里并未自动安装# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function check_nfscli(){ case $os_type in ubuntu|debian) if dpkg -l | grep -q nfs-common >/dev/null 2>&1; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] nfs-common 已经安装在主机上.${NC}" else echo -e "${GREEN}[INFO] nfs-common has been installed on the host.${NC}" fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] nfs-common 未安装在主机上, 请执行命令安装 'apt -y install nfs-common'.${NC}" else echo -e "${YELLOW}[WARN] nfs-common is not installed on the host, please execute the command install 'apt-get update && apt -y install nfs-common'.${NC}" fi fi ;; *) if rpm -qa | grep -q nfs-utils >/dev/null 2>&1; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] nfs-utils 已经安装在主机上.${NC}" else echo -e "${GREEN}[INFO] nfs-utils has been installed on the host.${NC}" fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] nfs-utils 未安装在主机上, 请执行命令安装 'yum -y install nfs-utils'.${NC}" else echo -e "${YELLOW}[WARN] nfs-utils is not installed on the host, please execute the command install 'yum -y install nfs-utils'.${NC}" fi fi ;; esac}function check_openssh(){ if ssh -V >/dev/null 2>&1; then OPENSSH_VERSION=$(ssh -V |& awk -F[_.] '{print $2}') if [ "${OPENSSH_VERSION}" -lt "7" ];then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] Openssh 版本必须高于 7.0.${NC}" else echo -e "${YELLOW}[WARN] Openssh version must be higher than 7.0 ${NC}" fi fi else if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${RED}[ERROR] 需要安装 7.0+ 版本的openssh.${NC}" exit 1 else echo -e "${RED}[ERROR] Need to install 7.0+ version of openssh.${NC}" exit 1 fi fi if grep -v "^\s*#" /etc/ssh/sshd_config | grep "AllowTcpForwarding yes" >/dev/null 2>&1; then if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${GREEN}[INFO] /etc/ssh/sshd_config 已经配置 AllowTcpForwarding yes.${NC}" else echo -e "${GREEN}[INFO] /etc/ssh/sshd_config has been configured AllowTcpForwarding yes.${NC}" fi else if grep "AllowTcpForwarding no" /etc/ssh/sshd_config >/dev/null 2>&1; then sed -i '/AllowTcpForwarding/s/^/#/' /etc/ssh/sshd_config sed -i '$a\AllowTcpForwarding yes' /etc/ssh/sshd_config else sed -i '$a\AllowTcpForwarding yes' /etc/ssh/sshd_config fi if [ "$LANG" == "zh_CN.UTF-8" ]; then echo -e "${YELLOW}[WARN] /etc/ssh/sshd_config 配置 AllowTcpForwarding yes 成功, 请执行命令重启 sshd 服务生效, 'systemctl restart sshd'.${NC}" else echo -e "${YELLOW}[WARN] /etc/ssh/sshd_config AllowTcpForwarding yes is successfully configured, Run the following command to restart the sshd service to take effect, 'systemctl restart sshd'.${NC}" fi fi}################################################ 描述：优化参数# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function optimize_linux() { sudo bash -c 'cat > /etc/sysctl.conf << EOFnet.bridge.bridge-nf-call-ip6tables=1net.bridge.bridge-nf-call-iptables=1net.ipv4.ip_forward=1net.ipv4.conf.all.forwarding=1net.ipv4.neigh.default.gc_thresh1=4096net.ipv4.neigh.default.gc_thresh2=6144net.ipv4.neigh.default.gc_thresh3=8192net.ipv4.neigh.default.gc_interval=60net.ipv4.neigh.default.gc_stale_time=120kernel.perf_event_paranoid=-1#sysctls for k8s node confignet.ipv4.tcp_slow_start_after_idle=0net.core.rmem_max=16777216fs.inotify.max_user_watches=524288kernel.softlockup_all_cpu_backtrace=1kernel.softlockup_panic=0kernel.watchdog_thresh=30fs.file-max=2097152fs.inotify.max_user_instances=8192fs.inotify.max_queued_events=16384vm.max_map_count=262144fs.may_detach_mounts=1net.core.netdev_max_backlog=16384net.ipv4.tcp_wmem=4096 12582912 16777216net.core.wmem_max=16777216net.core.somaxconn=32768net.ipv4.ip_forward=1net.ipv4.tcp_max_syn_backlog=8096net.ipv4.tcp_rmem=4096 12582912 16777216net.ipv6.conf.all.disable_ipv6=1net.ipv6.conf.default.disable_ipv6=1net.ipv6.conf.lo.disable_ipv6=1kernel.yama.ptrace_scope=0vm.swappiness=0kernel.core_uses_pid=1# Do not accept source routingnet.ipv4.conf.default.accept_source_route=0net.ipv4.conf.all.accept_source_route=0# Promote secondary addresses when the primary address is removednet.ipv4.conf.default.promote_secondaries=1net.ipv4.conf.all.promote_secondaries=1# Enable hard and soft link protectionfs.protected_hardlinks=1fs.protected_symlinks=1net.ipv4.conf.all.rp_filter=0net.ipv4.conf.default.rp_filter=0net.ipv4.conf.default.arp_announce = 2net.ipv4.conf.lo.arp_announce=2net.ipv4.conf.all.arp_announce=2net.ipv4.tcp_max_tw_buckets=5000net.ipv4.tcp_syncookies=1net.ipv4.tcp_fin_timeout=30net.ipv4.tcp_synack_retries=2kernel.sysrq=1EOF' sudo sysctl -p >/dev/null 2>&1 echo -e "${GREEN}[INFO] 优化kernel参数成功${NC}"}function optimize_limits() { sudo bash -c 'cat >> /etc/security/limits.conf <<EOF* soft nofile 1024000* hard nofile 1024000EOF' echo -e "${GREEN}[INFO] 优化limits参数成功${NC}"}function check_syscfg(){ sudo chmod 777 /etc/sysctl.conf sudo chmod 777 /sbin/sysctl sudo chmod 777 /etc/security/limits.conf optimize_linux optimize_limits sudo chmod 644 /etc/sysctl.conf sudo chmod 755 /sbin/sysctl sudo chmod 644 /etc/security/limits.conf}################################################ 描述： calico 网络配置初始化# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function check_network() { sudo bash -c 'cat >> /etc/NetworkManager/conf.d/calico.conf << EOF[keyfile]unmanaged-devices=interface-name:cali*;interface-name:tunl*unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:wireguard.caliEOF' systemctl restart NetworkManager}################################################ 描述：主入口函数# QingHub Studio官网: https://qinghub.net# 如过您安装遇到问题，请到官网查找官方联系方式或加支持群:# https://qinghub.net###############################################function main { echo -e "${GREEN}[INFO] ==========开始检查并配置初始化========= ${NC}" # 停止 friewalld disable_firewalld # 关闭 swap partition permanently disable_swap # 配置检查时间同步 check_time_sync # 配置安装时间同步组件 install_chrony # 配置检查 nfs-utils kubeadmin方式安装不需要检查 #check_nfscli # 配置检查内核版本 check_kernel # 配置检查资源情况 check_resource # 配置检查SSH check_openssh # 配置检查系统配置 check_syscfg # 转发 IPv4 check_forwarding_ipv4 # 配置检查Docker用户并添加ssh免密认证<authoirzed_keys> check_user # 配置检查Docker 容器运行时为Containerd时，不需要检查 #check_docker # 配置检查Docker用户权限容器运行时为Containerd时，不需要检查 #check_user_permission # 配置检查网络 check_network echo -e "${GREEN}[INFO] ==========成功完成检查并配置初始化========= ${NC}"}main

组件安装

需要安装的组件列表

安装containerd部署containerd 配置文件安装cni plugin安装crictl安装runc安装部署kubeadm、kubelet、kubectl

特别关注：

再拷贝kubeadm、kubelet、kubectl文件后，务必记得要创建必须要的几个目录，否则会导致后面初始化控制平面时会出错，详情见完整的安装脚本

sudo mkdir -p /var/lib/kubelet

sudo mkdir -p /etc/kubernetes/manifests

yum 自动从网络安装不会出现该问题，会自动创建。

初始化集群第一控制平面节点

安装完整的脚本

执行命令初始化第一个控制平面节点

在上节的安装过程中，实际以及包含了初始化第一个控制平面的脚本，由于其重要性，这里单独提出来详细说明。

通过执行kubeadm init 指令快速初始化控制平面，可以通过直接待参数的方式或通过–config加配置文件的方式实现：

配置参数形式

kubeadm init \ --apiserver-advertise-address="${QINGCLOUD_ADVERTISE_ADDRESS}" \ --image-repository "${LOCAL_IMAGE_DOMAIN}/${LOCAL_IMAGE_NAMESPACE}" \ --kubernetes-version "${KUBERNETES_VERSION}" \ --cri-socket=unix:///run/containerd/containerd.sock \ --service-cidr=10.96.0.0/16 \ --pod-network-cidr=172.16.0.0/16

配置文件形式

第一步：如下命令生成默认配置文件并作修改

kubeadm config print init-defaults > initConfig.yaml

第二步：执行命令

kubeadm init --config=initConfig.yaml

准备kubectl配置文件

根据上面的指令生成的结果，拷贝如下命令后直接执行，使得kubectl后面可以直接运行

mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config

加入Node节点

根据执行命令生成第一个控制平面的结果，拷贝如下类似的文件到node节点。直接执行：

kubeadm join x.x.x.x:6443 --token picxyk.su53y03l1z0jh333e \--discovery-token-ca-cert-hash sha256:01fecbd4bdc0513811f7b65a43e41820d703a91c631e262b1ad4634e12cd0205 --cri-socket=unix:///run/containerd/containerd.sock

执行kubectl get nodes 查看node节点执行结果

[root@itserver-master2 kube]# kubectl get nodesNAME STATUS ROLES AGE VERSIONitserver-master2 Ready control-plane 2h v1.29.4itserver-node4 Ready <none> 1m v1.29.4[root@itserver-master2 kube]#

部署网络插件

我们采用calico作为网络插件，calico最新版建议的部署方式为两个步骤，执行两个文件即可：

tigera-operator.yaml、custom-resources.yaml

下载tigera-operator.yaml

https://github.com/projectcalico/calico/blob/v3.27.3/manifests/tigera-operator.yaml

https://github.com/projectcalico/calico/blob/v3.27.3/manifests/custom-resources.yaml

修改tigera-operator.yaml文件内容（离线版）

注意：修改镜像地址，请根据自己保存镜像的实际地址修改

[root@web02 v1.29.4]# cat tigera-operator.yaml | grep image: image: image: quay.io/tigera/operator:v1.32.7[root@web02 v1.29.4]# [root@web02 v1.29.4]# sudo sed -i "s#quay.io\/tigera#qinghub.net:5000\/qingcloudtech#g" tigera-operator.yaml[root@web02 v1.29.4]# cat tigera-operator.yaml | grep image: image: image: qinghub.net:5000/qingcloudtech/operator:v1.32.7[root@web02 v1.29.4]# ll

执行kubectl create -f tigera-operator.yaml

[root@itserver-master2 kube]# kubectl create -f tigera-operator.yaml namespace/tigera-operator createdcustomresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io createdcustomresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io createdcustomresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io createdcustomresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io createdserviceaccount/tigera-operator createdclusterrole.rbac.authorization.k8s.io/tigera-operator createdclusterrolebinding.rbac.authorization.k8s.io/tigera-operator createddeployment.apps/tigera-operator created[root@itserver-master2 kube]#

查看结果

[root@itserver-master2 kube]# kubectl get pods -n tigera-operatorNAME READY STATUS RESTARTS AGEtigera-operator-6779dc6889-zd4zt 1/1 Running 0 55s[root@itserver-master2 kube]#

修改custom-resources.yaml

注意：主要修改内容：cidr: 172.16.0.0/16，需要与控制平面初始化时的地址填写的地址一直

spec: # Configures Calico networking. calicoNetwork: # Note: The ipPools section cannot be modified post-install. ipPools: - blockSize: 26 cidr: 172.16.0.0/12 encapsulation: VXLANCrossSubnet natOutgoing: Enabled nodeSelector: all()

执行kubectl create -f custom-resources.yaml

[root@itserver-master2 kube]# kubectl create -f custom-resources.yaml installation.operator.tigera.io/default createdapiserver.operator.tigera.io/default created[root@itserver-master2 kube]# kubectl get nsNAME STATUS AGEcalico-system Active 48sdefault Active 3h5mkube-node-lease Active 3h5mkube-public Active 3h5mkube-system Active 3h5mtigera-operator Active 6m35s[root@itserver-master2 kube]# kubectl get pods -n calico-systemNAME READY STATUS RESTARTS AGEcalico-kube-controllers-68bf945ffc-mf7t2 0/1 ContainerCreating 0 75scalico-node-27fgm 0/1 Init:ImagePullBackOff 0 75scalico-typha-5886b45b65-pmsm7 0/1 ErrImagePull 0 75scsi-node-driver-9b29j 0/2 ContainerCreating 0 75s[root@itserver-master2 kube]#

calico网络安装后，检查所有空间众的pod：

[root@itserver-master2 certs.d]# kubectl get pods --all-namespacesNAMESPACE NAME READY STATUS RESTARTS AGEcalico-apiserver calico-apiserver-864697c659-2sdhd 1/1 Running 0 4m18scalico-apiserver calico-apiserver-864697c659-c2vp9 1/1 Running 0 4m18scalico-system calico-kube-controllers-68bf945ffc-dvrlf 1/1 Running 0 63mcalico-system calico-node-27fgm 1/1 Running 0 18hcalico-system calico-node-zwpls 1/1 Running 0 17hcalico-system calico-typha-5886b45b65-pmsm7 1/1 Running 0 18hcalico-system csi-node-driver-9b29j 2/2 Running 0 18hcalico-system csi-node-driver-mrtq5 2/2 Running 0 17hkube-system coredns-67bd986d4c-67fvl 1/1 Running 0 16mkube-system coredns-67bd986d4c-x7vk7 1/1 Running 0 56mkube-system etcd-itserver-master2 1/1 Running 1 21hkube-system kube-apiserver-itserver-master2 1/1 Running 1 21hkube-system kube-controller-manager-itserver-master2 1/1 Running 1 21hkube-system kube-proxy-9rv85 1/1 Running 0 21hkube-system kube-proxy-l9rht 1/1 Running 1 17hkube-system kube-scheduler-itserver-master2 1/1 Running 1 21htigera-operator tigera-operator-6779dc6889-zd4zt 1/1 Running 0 18h[root@itserver-master2 certs.d]#

看到如上几个空间中的状态都变为runging时，网络部署成功

测试验证

部署nginx

第一步、部署文件准备nginx.yaml

apiVersion: v1kind: Servicemetadata: name: nginx labels: app: nginxspec: ports: - port: 80 name: web selector: app: nginx---apiVersion: apps/v1kind: Deploymentmetadata: name: nginx-deploymentspec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: qingcloudtech/nginx:1.25.4 ports: - containerPort: 80 name: web

第二部、执行命令

kubectl apply -f nginx.yaml

第三步、检查service及deployment

[root@itserver-master2 kube]# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEkubernetes ClusterIP 10.96.0.1 <none> 443/TCP 22hnginx ClusterIP 10.101.221.253 <none> 80/TCP 17s[root@itserver-master2 kube]# kubectl get deploy -o wideNAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTORnginx-deployment 1/1 1 1 35s nginx qingcloudtech/nginx:1.25.4 app=nginx[root@itserver-master2 kube]# [root@itserver-master2 kube]# curl http://172.24.128.130<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>html { color-scheme: light dark; }body { width: 35em; margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif; }</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed andworking. Further configuration is required.</p><p>For online documentation and support please refer to<a href="http://nginx.org/">nginx.org</a>.<br/>Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p></body></html>[root@itserver-master2 kube]#

至此，完成整个部署过程

你可以通过【QingHub Studio】) 套件直接安装部署，也可以手动按如下文档操作，该项目已经全面开源，完整的脚本可以从如下开源地址获取：

开源地址: https://gitee.com/qingplus/qingcloud-platform

【QingHub Studio集成开发套件】

上一篇： nginx（NGINX）详细下载安装及使用教程

下一篇： nginx 手动离线安装及依赖包安装(pcre+zlib+openssl)

本文标签

【k8s】Kubernetes 1.29.4离线安装部署（总）

声明

本文内容仅代表作者观点，或转载于其他网站，本站不以此文作为商业用途
如有涉及侵权，请联系本站进行删除
转载本站原创文章，请注明来源及作者。