「Rancher」- 常见问题处理

  CREATED BY JENKINSBOT

修改日志等级

Rancher Docs: Logging

# 通过 Docker 部署
docker exec -ti <container_id> loglevel --set debug  # debug info trace

[ERROR] error syncing ‘system-library’

...
rancher-server    | 2022/08/01 09:47:03 [ERROR] error syncing 'system-library': handler system-image-upgrade-catalog-controller: upgrade cluster c-m-dqgvppq8 system service alerting failed: template system-library-rancher-monitoring incompatible with rancher version or cluster's [c-m-dqgvppq8] kubernetes version, requeuing
rancher-server    | 2022/08/01 09:47:03 [ERROR] error syncing 'system-library': handler system-image-upgrade-catalog-controller: upgrade cluster c-m-dqgvppq8 system service alerting failed: template system-library-rancher-monitoring incompatible with rancher version or cluster's [c-m-dqgvppq8] kubernetes version, requeuing
...

原因分析:
原始的 Catelog system-library 是 Rancher 自带的,使用 release-v2.6 分支;

解决方案:
修改系统 Catelog 配置,将 system-library 的 branche 设置为 master;

部署 Helm 应用失败;helm-operation … ErrImagePull …

问题描述:在 Rancher 中,通过 Helm 部署应用失败;

原因分析:我们看到 helm-operation-6dvgh 拉取镜像失败,我们猜测是:helm operation Pod 负责 Helm 部署,而镜像拉取失败导致出现该错误;

# kubectl get pods -n cattle-system
NAME                               READY   STATUS         RESTARTS   AGE
cm-acme-http-solver-4r5n4          1/1     Running        0          7d7h
helm-operation-6dvgh               0/2     ErrImagePull   0          51m
rancher-5ddfb86964-2d9kt           1/1     Running        0          17d
rancher-5ddfb86964-98bpc           1/1     Running        0          17d
rancher-5ddfb86964-j7q4z           1/1     Running        0          17d
rancher-webhook-565d58fffd-rjkk6   1/1     Running        0          17d

# kubectl describe pod helm-operation-6dvgh
...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  45m                default-scheduler  Successfully assigned cattle-system/helm-operation-6dvgh to cn-hangzhou.172.18.3.203
  Warning  Failed     34m                kubelet            Error: ImagePullBackOff
  Warning  Failed     34m                kubelet            Failed to pull image "rancher/shell:v0.1.18": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/shell:v0.1.18": failed to copy: read tcp 172.18.3.203:43858->104.18.121.25:443: read: connection reset by peer
  Normal   BackOff    34m                kubelet            Back-off pulling image "rancher/shell:v0.1.18"
  Warning  Failed     11m (x2 over 34m)  kubelet            Error: ErrImagePull
  Normal   BackOff    11m (x3 over 34m)  kubelet            Back-off pulling image "rancher/shell:v0.1.18"
  Warning  Failed     11m (x3 over 34m)  kubelet            Error: ImagePullBackOff
  Warning  Failed     11m                kubelet            Failed to pull image "rancher/shell:v0.1.18": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/shell:v0.1.18": failed to copy: read tcp 172.18.3.203:45852->104.18.125.25:443: read: connection reset by peer
  Normal   Pulling    11m (x3 over 45m)  kubelet            Pulling image "rancher/shell:v0.1.18"
...

解决方案:我们尝试修改 rancher/shell 镜像的地址:

// 我们尝试修改 Helm Chart values.yaml 文件
// postDelete.image.repository,但似乎并未生效,从参数名来看也与该参数无关;

// 根据社区反馈:https://forums.rancher.com/t/how-can-i-change-the-rancher-shell-image/36630
// 应该修改 CR 的配置

# kubectl edit settings.management.cattle.io
...
  default: rancher/shell:v0.1.10
...