问题描述
该笔记将记录:在 Kubernetes 中,如何部署 Rook 服务,底层使用 NFS 存储,以及常见问题解决方案;
解决方案
Rook NFS v1.7(03/14/2022),建议阅读 NFS Docs/v1.7 文档以了解更多细节,这里我们仅记录适用于我们测试环境的部署过程。
Kubernetes HA Cluster 1.18.20, worker k8s-storage as dedicated storage node
环境要求
Kubernetes v1.16 or higher
The desired volume to export needs to be attached to the NFS server pod via a PVC
NFS client packages must be installed on all nodes where Kubernetes might run pods with NFS mounted.
关于存储:
1)简单的拓扑结构为 Normal Pod ⇒ Storage Class ⇒ NFS Server ⇒ PVC ⇒ PV (hostPath) 所以我们以 hostPath 方式来提供最终的存储;
2)通过专用的存储节点,即 Kubernetes Worker 但是不会向该节点调度 Pod 实例(通过 Taint 及 Namespace defaultTolerations 来实现);
准备工作
# Taint node,以专用于存储 kubectl taint nodes k8s-storage dedicated=storage:NoSchedule # 开启 PodNodeSelector,PodTolerationRestriction 插件(不再细述) kube-apiserver ... --enable-admission-plugins=NodeRestriction,PodNodeSelector,PodTolerationRestriction ...
STEP-01 Deploy NFS Operator
git clone --single-branch --branch v1.7.3 https://github.com/rook/nfs.git cd nfs/cluster/examples/kubernetes/nfs kubectl create -f crds.yaml kubectl create -f operator.yaml # kubectl get pods -n rook-nfs-system NAME READY STATUS RESTARTS AGE rook-nfs-operator-794b5c98bd-rc8lv 1/1 Running 0 8m31s
补充说明:
1)Operator 是否调度到 k8s-storage(专用存储节点)并不重要;
STEP-02 Create and Initialize NFS Server
kubectl apply -f ./01-rbac.yaml
kubectl apply -f ./02-nfs-server.yaml
查看结果:
# kubectl -n rook-nfs get nfsservers.nfs.rook.io NAME AGE STATE rook-nfs 2m Running # kubectl -n rook-nfs get pod -l app=rook-nfs -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rook-nfs-0 2/2 Running 0 6m8s 192.168.59.130 k8s-w03 <none> <none>
STEP-03 Accessing the Export
kubectl apply -f ./03-storage-class.yaml
kubectl apply -f ./testing.yaml
补充说明
Pod 通过 Service 进行 NFS 挂载:
# kubectl -n rook-nfs get service rook-nfs NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE rook-nfs ClusterIP 10.111.156.207 <none> 2049/TCP,111/TCP 135m
调试追踪
# kubectl -n rook-nfs-system logs -l app=rook-nfs-operator # kubectl -n rook-nfs logs rook-nfs-0 nfs-server # NFS Server # kubectl -n rook-nfs logs rook-nfs-0 nfs-provisioner # Storage Class
参考文献
Default Toleration at Namespace Level | by Zhimin Wen | Medium
DaemonSet not respecting Namespace defaultTolerations · Issue #94722 · kubernetes/kubernetes
Taints and Tolerations | Kubernetes
plugins – k3s node restriction for namespace – Stack Overflow
Rook NFS/v1.7.3/Network Filesystem (NFS) Quickstart