「Kubernetes」- 使用 containerd 环境、切换 Docker 为 containerd 环境

  CREATED BY JENKINSBOT

问题描述

# 04/18/2022 在 Kubernetes Cluster 中，我们又尝试使用 containerd 作为容器环境。

该笔记将记录：在 Kubernetes Cluster 中使用 containerd 的方法，以及相关问题的解决办法。

解决方案

阅读 Changing the Container Runtime on a Node from Docker Engine to containerd 文档，以获取容器运行时替换的更多细节。

v1.22.10-aliyun.1 => containerd://1.5.13

第零步、准备工作

kubectl drain --ignore-daemonsets --delete-emptydir-data <node-to-drain>

SSH> systemctl stop kubelet && systemctl disable docker.service --now

第一步、配置 containerd 服务

配置 Kernel 参数

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# Apply sysctl params without reboot
sysctl --system

部署 containerd 服务

CRI/containerd

Kubernetes 1.22, containerd 1.5.13
Kubernetes 1.24, containerd 1.5.13

调整 containerd 参数

# vim /etc/containerd/config.toml
...

# 通过 Docker 仓库安装 continerd 服务，需要注释该行
# 否则 kubeadm join 失败：[ERROR CRI]: container runtime is not running: ...
# https://github.com/containerd/containerd/issues/4581
# disabled_plugins = ["cri"]                                                    

...

# 使用 systemd cgroup 驱动
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

# 配置 sandbox 镜像地址，参考
# https://github.com/kubernetes/kubeadm/issues/1610
# https://github.com/kubernetes/kubernetes/issues/62732
# https://github.com/containerd/cri/blob/master/docs/config.md
[plugins.cri]
  # sandbox_image is the image used by sandbox container.
  sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.2"

...

# systemctl restart containerd && systemctl enable containerd

关于容器日志轮转问题

// 日志轮转由 kubelet 负责处理，参考文献：
// https://github.com/containerd/containerd/issues/4830
// https://github.com/kubernetes/kubernetes/pull/59898
// https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level
// https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration
// containerLogMaxSize.default==10Mi, containerLogMaxFiles.default==5

第二步、切换 containerd 环境

参考 Changing the Container Runtime on a Node from Docker Engine to containerd 文档，以获取将 Docker 切换为 contaderd.io 的官方说明。 

SSH> vim /var/lib/kubelet/kubeadm-flags.env
... --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock ...

# kubectl edit node <node-name>
...
metadata:
  annotations:
    kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
...

SSH> systemctl start kubelet

第三步、节点：测试、清理、恢复

SSH> apt-get purge -y docker-ce*

# 理论上不需要重启节点，但是需要确保旧容器完全停止，
# 否则，新旧运行时切换导致 kubectl 会重新创建容器，否则新旧容器会产生冲突（例如端口占用等等）
SSH> reboot 

kubectl uncordon <node-name>
kubectl get nodes -o wide

参考文献

Changing the Container Runtime on a Node from Docker Engine to containerd | Kubernetes
Container runtimes | Kubernetes