「Rook」- NFS（实验目的，云原生 NFS 存储）

  CREATED BY JENKINSBOT

问题描述

该笔记将记录：在 Kubernetes 中，如何部署 Rook 服务，底层使用 NFS 存储，以及常见问题解决方案；

解决方案

Rook NFS v1.7（03/14/2022），建议阅读 NFS Docs/v1.7 文档以了解更多细节，这里我们仅记录适用于我们测试环境的部署过程。
Kubernetes HA Cluster 1.18.20, worker k8s-storage as dedicated storage node

环境要求

Kubernetes v1.16 or higher
The desired volume to export needs to be attached to the NFS server pod via a PVC
NFS client packages must be installed on all nodes where Kubernetes might run pods with NFS mounted.

关于存储：
1）简单的拓扑结构为 Normal Pod ⇒ Storage Class ⇒ NFS Server ⇒ PVC ⇒ PV (hostPath) 所以我们以 hostPath 方式来提供最终的存储；
2）通过专用的存储节点，即 Kubernetes Worker 但是不会向该节点调度 Pod 实例（通过 Taint 及 Namespace defaultTolerations 来实现）；

准备工作

# Taint node，以专用于存储
kubectl taint nodes k8s-storage dedicated=storage:NoSchedule

# 开启 PodNodeSelector,PodTolerationRestriction 插件（不再细述）
kube-apiserver ... --enable-admission-plugins=NodeRestriction,PodNodeSelector,PodTolerationRestriction ...

STEP-01 Deploy NFS Operator

git clone --single-branch --branch v1.7.3 https://github.com/rook/nfs.git
cd nfs/cluster/examples/kubernetes/nfs
kubectl create -f crds.yaml
kubectl create -f operator.yaml

# kubectl get pods -n rook-nfs-system 
NAME                                 READY   STATUS    RESTARTS   AGE
rook-nfs-operator-794b5c98bd-rc8lv   1/1     Running   0          8m31s

补充说明：
1）Operator 是否调度到 k8s-storage（专用存储节点）并不重要；

STEP-02 Create and Initialize NFS Server

kubectl apply -f ./01-rbac.yaml
kubectl apply -f ./02-nfs-server.yaml

查看结果： 

# kubectl -n rook-nfs get nfsservers.nfs.rook.io
NAME       AGE   STATE
rook-nfs   2m    Running

# kubectl -n rook-nfs get pod -l app=rook-nfs -o wide
NAME         READY   STATUS    RESTARTS   AGE    IP               NODE      NOMINATED NODE   READINESS GATES
rook-nfs-0   2/2     Running   0          6m8s   192.168.59.130   k8s-w03   <none>           <none>

STEP-03 Accessing the Export

kubectl apply -f ./03-storage-class.yaml
kubectl apply -f ./testing.yaml

补充说明

Pod 通过 Service 进行 NFS 挂载： 

# kubectl -n rook-nfs get service rook-nfs 
NAME       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)            AGE
rook-nfs   ClusterIP   10.111.156.207   <none>        2049/TCP,111/TCP   135m

调试追踪

# kubectl -n rook-nfs-system logs -l app=rook-nfs-operator

# kubectl -n rook-nfs logs rook-nfs-0 nfs-server                                # NFS Server

# kubectl -n rook-nfs logs rook-nfs-0 nfs-provisioner                           # Storage Class

参考文献

Default Toleration at Namespace Level | by Zhimin Wen | Medium
DaemonSet not respecting Namespace defaultTolerations · Issue #94722 · kubernetes/kubernetes
Taints and Tolerations | Kubernetes
plugins – k3s node restriction for namespace – Stack Overflow
Rook NFS/v1.7.3/Network Filesystem (NFS) Quickstart