【Kubernetes】k8s使用minio作为对象存储
白山茶AI 2024-09-04 09:07:14 阅读 58
一、部署minio operator
k8s version：v1.20.15 minio version ：v4.4.16
（1）安装kubectl-minio插件
自选minio-operaterd的版本下载包
minio-operater plugin
<code># 上传服务器并解压
unzip kubectl-minio_linux_amd64.zip "kubectl-minio" -d /usr/local/bin/
# 查看是否安装成功。如返回版本信息则安装成功
kubectl minio version
# 初始化
kubectl minio init
# operator默认部署到minio-operator命名空间中，如果需要指定命名空间，可使用kubectl minio init --namespace {YOUR-NAMESPACE}。
# 使用群集。在配置operator的DNS主机名时，将本地作为群集域。指定kubectl minio init --cluster域参数以设置不同的集群域值。
# 验证是否安装成功
[root@k8s-master-4 ~]# kubectl get all -n minio-operator
NAME READY STATUS RESTARTS AGE
pod/console-5f4f574656-5s2wx 1/1 Running 0 5h52m
pod/minio-operator-c78cb4c65-875rt 1/1 Running 0 5h52m
pod/minio-operator-c78cb4c65-x4869 1/1 Running 0 5h52m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/console ClusterIP 10.98.66.116 <none> 9090/TCP,9443/TCP 5h52m
service/operator ClusterIP 10.96.133.155 <none> 4222/TCP,4221/TCP 5h52m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/console 1/1 1 1 5h52m
deployment.apps/minio-operator 2/2 2 2 5h52m
NAME DESIRED CURRENT READY AGE
replicaset.apps/console-5f4f574656 1 1 1 5h52m
replicaset.apps/minio-operator-c78cb4c65 2 2 2 5h52m
# 访问Operator Console，新开一个终端窗口
[root@k8s-master-4 ~]# kubectl minio proxy -n minio-operator
Starting port forward of the Console UI.
To connect open a browser and go to http://localhost:9090
Current JWT to login: eyJhbGciOiJSUzI1NiIsImtpZCI6Ii1MY0QybWl6YklaUzE1MHZQNzcxcFNpc2JrUk1NdGU1X1MzaHUtY0hmMWcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXRva2VuLThwcTRtIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNvbnNvbGUtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIyMzY4ODJjZS1hMDhhLTQ0MzEtOWY5MC1iN2IxMWJkZWJmMDEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bWluaW8tb3BlcmF0b3I6Y29uc29sZS1zYSJ9.VpR7HaUT6AU_u61AgGAe5thnXoWcPP7nSK2S7zorUXVShdKfNIO1pdvTz1XJ9X1b7hlk3Yqn6LfLz1WJXxDBhEz_pHSqJ0s-RCv00hEHGRLPzyUdI-s9zjOwwqU7yITxkyNswodiZ6jRCOiMwqPRZEHKetzMZlckK6xvIO6gpcQd0dDwXagNyHGArb8-zvBLzqWhgx0PoB3B-FHuy7VabGzVDoFdMQiIU20KOjvdQEXPPZRfk-RBMM-hy7HdCM-rk4sTYEhTGz9dUzj1_umvQbupZw7kEcw588MMF9Q2Q66hOCOC-6YbQTPJIHzqwuYNv4SOzkj5hds2eUi_1FHHDA
Forwarding from 0.0.0.0:9090 -> 9090
Handling connection for 9090
# JWT的值为后续登陆界面时使用
访问地址：http://ip:9090
二、创建StorageClass
1、sc-minio.yaml
<code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: minio-local-storage # SC-NAME
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
创建
kubectl apply -f sc-minio.yaml
# 查看是否创建成功
[root@k8s-master-4 ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
minio-local-storage kubernetes.io/no-provisioner Retain WaitForFirstConsumer false 5h38m
2、创建所需的永久卷
| 序号 | 路径 | 容量 (G) | 说明 | 所在节点 |
|---|---|---|---|---|
| 1 | /data/1 | 5 | 租户使用 | 3个节点各1个 |
| 2 | /data/log1 | 5 | 租户使用 | 3个节点各1个 |
| 3 | /data/minio-tenant-1-log-0 | 5 | 审计日志 | k8s-master-4 |
| 4 | /data/minio-tenant-1-prometheus | 5 | 监控数据 | k8s-master-4 |
租户：30G
<code># 创建目录(三个节点)
mkdir -p /data/1 /data/log1
# 在K8s-master-4创建审计日志目录、监控目录
mkdir -p /data/minio-tenant-1-log-0 /data/minio-tenant-1-prometheus
三、PV && PVC
为MinIO租户中的每个卷创建一个PV。例如，假设一个Kubernetes集群有3个节点，每个节点有2个本地连接的驱动器，那么总共创建6个本地PV。名称、容量大小、目录等可根据个人环境进行配置:
1、pv-tenant.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-k8s-master-4-data1 # PV-NAME
spec:
capacity:
storage: 5Gi # capacity
volumeMode: Filesystem
accessModes:
- ReadWriteOnce # default ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: minio-local-storage # SC-NAME
local:
path: /data/1 # SC local-path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-master-4 # NODE-NAME
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-k8s-master-4-log1 # PV-NAME
spec:
capacity:
storage: 5Gi # capacity
volumeMode: Filesystem
accessModes:
- ReadWriteOnce # default ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: minio-local-storage # SC-NAME
local:
path: /data/log1 # SC local-path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-master-4 # NODE-NAME
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-k8s-node1-5-data1 # PV-NAME
spec:
capacity:
storage: 5Gi # capacity
volumeMode: Filesystem
accessModes:
- ReadWriteOnce # default ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: minio-local-storage # SC-NAME
local:
path: /data/1 # SC local-path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-node1-5 # NODE-NAME
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-k8s-node1-5-log1 # PV-NAME
spec:
capacity:
storage: 5Gi # capacity
volumeMode: Filesystem
accessModes:
- ReadWriteOnce # default ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: minio-local-storage # SC-NAME
local:
path: /data/log1 # SC local-path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-node1-5 # NODE-NAME
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-k8s-node2-6-data1 # PV-NAME
spec:
capacity:
storage: 5Gi # capacity
volumeMode: Filesystem
accessModes:
- ReadWriteOnce # default ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: minio-local-storage # SC-NAME
local:
path: /data/1 # SC local-path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-node2-6 # NODE-NAME
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-k8s-node2-6-log1 # PV-NAME
spec:
capacity:
storage: 5Gi # capacity
volumeMode: Filesystem
accessModes:
- ReadWriteOnce # default ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: minio-local-storage # SC-NAME
local:
path: /data/log1 # SC local-path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-node2-6 # NODE-NAME
创建审计日志PV、PVC配置文件(后续备用)：
2、pv-pvc-minio-tenant-1-log-0.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-tenant-1-log # PV-NAME
namespace: minio-tenant-1
spec:
capacity:
storage: 5Gi # capacity
volumeMode: Filesystem
accessModes:
- ReadWriteOnce # default ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: minio-local-storage # SC-NAME
local:
path: /data/minio-tenant-1-log-0 # SC local-path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-master-4 # NODE-NAME
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
v1.min.io/log-pg: minio-tenant-1-log
name: minio-tenant-1-log-minio-tenant-1-log-0
namespace: minio-tenant-1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "5368709120"
storageClassName: minio-local-storage
volumeMode: Filesystem
volumeName: minio-tenant-1-log
创建租户监控prometheus使用的PV、PVC配置文件(后续备用)：
3、pv-pvc-minio-tenant-prometheus.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-tenant-1-prometheus # PV-NAME
namespace: minio-tenant-1
spec:
capacity:
storage: 5Gi # capacity
volumeMode: Filesystem
accessModes:
- ReadWriteOnce # default ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: minio-local-storage # SC-NAME
local:
path: /data/minio-tenant-1-prometheus # SC local-path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s3-master # NODE-NAME
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
v1.min.io/prometheus: minio-tenant-1-prometheus
name: minio-tenant-1-prometheus-minio-tenant-1-prometheus-0
namespace: minio-tenant-1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: minio-local-storage
volumeMode: Filesystem
volumeName: minio-tenant-1-prometheus
# 为MinIO Tenant创建命名空间
kubectl create namespace minio-tenant-1
# 创建租户pvc
kubectl apply -f pv-tenant.yaml
# 查看创建的pv
kubectl get pv | grep minio
四、创建MinIO Tenant租户
使用kubectl minio租户创建命令创建minio租户。该命令始终使用MinIO服务器和MinIO控制台的最新稳定Docker映像。
以下示例创建了一个3节点MinIO部署，6个驱动器的总容量为30Gi。此时会同时创建审计日志、prometheus监控相关pod。
<code># 查看minio支持的api-resource版本
[root@k8s-master-4 ~]# kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
......
tenants tenant minio.min.io/v2 true Tenant
......
[root@k8s-master-4 ~]# kubectl minio tenant create minio-tenant-1 \
> --servers 3 \
> --volumes 6 \
> --capacity 30Gi \
> --storage-class minio-local-storage \
> --namespace minio-tenant-1
Tenant 'minio-tenant-1' created in 'minio-tenant-1' Namespace
Username: W175STC2EU3QG0KXEEUN
Password: BhbYGhIZamMlYQXjYOZaD8TZzJno9zNwXa0VUmrq
Note: Copy the credentials to a secure location. MinIO will not display these again.
APPLICATION SERVICE NAME NAMESPACE SERVICE TYPE SERVICE PORT
MinIO minio minio-tenant-1 ClusterIP 443
Console minio-tenant-1-console minio-tenant-1 ClusterIP 9443
# 查看租户状态
# 当前状态在初始化过程中，等待Log Search就绪，此过程可能要花费一段时间，租户成功后，状态会显示Initialized：
[root@k8s-master-4 ~]# kubectl get tenants -n minio-tenant-1
NAME STATE AGE
minio-tenant-1 Initialized 19h
# 各pod、服务等对象成功状态
kubectl -n minio-tenant-1 get all
# 查看创建的pvc（如图二）
kubectl -n minio-tenant-1 get pvc
# 查看当前租户
[root@k8s-master-4 ~]# kubectl minio tenant list
Tenant 'minio-tenant-1', Namespace 'minio-tenant-1', Total capacity 30 GiB
Current status: Initialized
MinIO version: minio/minio:RELEASE.2022-04-16T04-26-02Z
# 可以使用kubectl端口转发程序临时公开每个服务。运行以下示例将流量从运行kubectl的主机转发到Kubernetes集群内运行的服务。
# 租户端口转发
[root@k8s-master-4 ~]# kubectl port-forward service/minio 443:443 -n minio-tenant-1
Forwarding from 127.0.0.1:443 -> 9000
Forwarding from [::1]:443 -> 9000
[root@k8s-master-4 ~]# kubectl port-forward service/minio-tenant-1-console 9443:9443 -n minio-tenant-1
Forwarding from 127.0.0.1:9443 -> 9443
现在我们进入minio的控制台查看
租户状态已经ok了
五、其他操作
<code># 删除minio-operater
kubectl minio delete
# 删除tenant
kubectl minio tenant delete minio-tenant-1 --namespace minio-tenant-1
# 删除pvc、pv
kubectl delete pvc --all -n minio-tenant-1
kubectl delete pv --all -n minio-tenant-1
# 删除命名空间
kubectl delete ns minio-tenant-1
声明
本文内容仅代表作者观点，或转载于其他网站，本站不以此文作为商业用途
如有涉及侵权，请联系本站进行删除
转载本站原创文章，请注明来源及作者。