「Kubernetes」- kube-controller-manager-k8scp-01

  CREATED BY JENKINSBOT

当将节点加入集群时,主节点的 kube-controller-manager-k8scp-01 发生重启。日志如下:

...
I0408 10:24:03.808759       1 shared_informer.go:247] Caches are synced for taint 
I0408 10:24:03.809161       1 node_lifecycle_controller.go:1429] Initializing eviction metric for zone: 
I0408 10:24:03.810221       1 taint_manager.go:187] Starting NoExecuteTaintManager
I0408 10:24:03.813179       1 event.go:291] "Event occurred" object="k8scp-02" kind="Node" apiVersion="v1" type="Normal" reason="RegisteredNode" message="Node k8scp-02 event: Registered Node k8scp-02 in Controller"
W0408 10:24:03.863942       1 node_lifecycle_controller.go:1044] Missing timestamp for Node k8scp-02. Assuming now as a timestamp.
W0408 10:24:03.864545       1 node_lifecycle_controller.go:1044] Missing timestamp for Node k8scp-01. Assuming now as a timestamp.
I0408 10:24:03.864623       1 node_lifecycle_controller.go:1245] Controller detected that zone  is now in state Normal.
I0408 10:24:03.864502       1 event.go:291] "Event occurred" object="k8scp-01" kind="Node" apiVersion="v1" type="Normal" reason="RegisteredNode" message="Node k8scp-01 event: Registered Node k8scp-01 in Controller"
I0408 10:24:03.891093       1 shared_informer.go:247] Caches are synced for HPA 
I0408 10:24:03.897319       1 shared_informer.go:247] Caches are synced for resource quota 
I0408 10:24:03.953374       1 shared_informer.go:247] Caches are synced for attach detach 
I0408 10:24:04.133022       1 shared_informer.go:240] Waiting for caches to sync for garbage collector
I0408 10:24:04.335804       1 shared_informer.go:247] Caches are synced for garbage collector 
I0408 10:24:04.400566       1 shared_informer.go:247] Caches are synced for garbage collector 
I0408 10:24:04.400653       1 garbagecollector.go:151] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0408 10:24:04.729922       1 request.go:655] Throttling request took 1.049063919s, request: GET:https://172.31.253.61:6443/apis/events.k8s.io/v1beta1?timeout=32s
I0408 10:24:05.582833       1 shared_informer.go:240] Waiting for caches to sync for resource quota
I0408 10:24:05.582903       1 shared_informer.go:247] Caches are synced for resource quota 
W0408 10:24:32.876272       1 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Event ended with: Internal error occurred: etcdserver: no leader
E0408 10:24:36.431506       1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: etcdserver: request timed out
E0408 10:24:39.417315       1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Get "https://172.31.253.61:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manag
er?timeout=10s": context deadline exceeded
I0408 10:24:39.417501       1 leaderelection.go:278] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition
F0408 10:24:39.417576       1 controllermanager.go:294] leaderelection lost
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00011e001, 0xc000a21440, 0x4c, 0x222)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x6f9f860, 0xc000000003, 0x0, 0x0, 0xc000b40540, 0x6dfe118, 0x14, 0x126, 0x0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x19b
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printf(0x6f9f860, 0x3, 0x0, 0x0, 0x0, 0x0, 0x47d0b1b, 0x13, 0x0, 0x0, ...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:750 +0x191
k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatalf(...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1502
k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2()
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:294 +0x8f
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1(0xc00101e6c0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:199 +0x29
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc00101e6c0, 0x4e656e0, 0xc000115400)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:209 +0x15d
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie(0x4e65720, 0xc00011c018, 0x4e9b3c0, 0xc0000b8a00, 0x37e11d600, 0x2540be400, 0x77359400, 0xc00103f280, 0x494a528, 0x0, ...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:222 +0x9c
...

从日志中,我们发现是 etcd 服务无法访问,而导致的失败。