Kubernetes 容器运行时网络未就绪
Kubernetes Container runtime network not ready
我安装了三个节点的 Kubernetes 集群，控制节点看起来没问题，当我尝试加入其他两个节点时，两个节点的状态都是：未就绪
在控制节点上：
[root@alva-anstawx01 /]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
alva-anstawx01.<domain name> Ready control-plane,master 7d20h v1.21.1
alva-anstawx02.<domain name> NotReady <none> 22h v1.21.1
alva-anstawx03.<domain name> NotReady <none> 22h v1.21.1
PODs 看起来还不错 运行:
NAME READY STATUS RESTARTS AGE
coredns-558bd4d5db-c8p97 1/1 Running 0 7d20h
coredns-558bd4d5db-hjb8g 1/1 Running 0 7d20h
etcd-alva-anstawx01.alv.autoliv.int 1/1 Running 2 7d20h
kube-apiserver-alva-anstawx01.alv.autoliv.int 1/1 Running 2 7d20h
kube-controller-manager-alva-anstawx01.alv.autoliv.int 1/1 Running 2 7d20h
kube-proxy-b8ft2 1/1 Running 0 7d20h
kube-proxy-frr7c 1/1 Running 0 23h
kube-proxy-ztxbf 1/1 Running 0 23h
kube-scheduler-alva-anstawx01.alv.autoliv.int 1/1 Running 2 7d20h
进一步检查日志，发现缺少某些内容，因此 CNI 插件在这些节点上启动，但不确定如何继续：
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 09 Jun 2021 12:24:34 +0200 Tue, 08 Jun 2021 14:00:45 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 09 Jun 2021 12:24:34 +0200 Tue, 08 Jun 2021 14:00:45 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 09 Jun 2021 12:24:34 +0200 Tue, 08 Jun 2021 14:00:45 +0200 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Wed, 09 Jun 2021 12:24:34 +0200 Tue, 08 Jun 2021 14:00:45 +0200 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
我在每个节点上都有一个网络接口
在受影响的节点上：
Jun 09 12:34:19 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:19.209657 1026 kubelet.go:2211] "Container runtime network not ready" networkReady="N
Jun 09 12:34:19 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:19.698034 1026 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"Sta
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817375 1026 remote_image.go:114] "PullImage from image service failed" err="rpc er
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817429 1026 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code =
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817627 1026 kuberuntime_manager.go:864] container &Container{Name:calico-typha,Ima
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817706 1026 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"Sta
Jun 09 12:34:24 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:24.211195 1026 kubelet.go:2211] "Container runtime network not ready" networkReady="N
我使用了 Calico 默认配置并且我在每个节点上都有一个接口：
控制节点：
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:ac:64:8c brd ff:ff:ff:ff:ff:ff
inet 10.4.9.73/21 brd 10.4.15.255 scope global ens192
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feac:648c/64 scope link
valid_lft forever preferred_lft forever
3: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 66:c5:7d:06:e5:fe brd ff:ff:ff:ff:ff:ff
inet 192.168.228.192/32 scope global vxlan.calico
valid_lft forever preferred_lft forever
inet6 fe80::64c5:7dff:fe06:e5fe/64 scope link
valid_lft forever preferred_lft forever
4: cali5441eeb56bd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
5: cali389c5f98ecc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
6: calicc306a285eb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
其他节点：
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:ac:67:61 brd ff:ff:ff:ff:ff:ff
inet 10.4.9.80/21 brd 10.4.15.255 scope global ens192
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feac:6761/64 scope link
valid_lft forever preferred_lft forever
谁能帮我解决如何在其他两个节点上初始化 calico 的问题？
编辑：
我已经解决了 /var space 的问题，但没有帮助：
[root@alva-anstawx03 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 60M 1.8G 4% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/vg_system-lv_root 9.8G 2.1G 7.2G 23% /
/dev/sda1 976M 206M 704M 23% /boot
/dev/mapper/vg_system-lv_var 19G 1.1G 17G 6% /var
/dev/mapper/vg_system-lv_opt 3.9G 72M 3.6G 2% /opt
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/77b1f7f0-8598-4640-af2f-e960c4c76120/volumes/kubernetes.io~projected/kube-api-access-7xnp8
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/4398eeeb-0f74-477c-a066-403ecab4abe1/volumes/kubernetes.io~projected/kube-api-access-9bh4m
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/56cbc698b06f57937128eadc74cc098c4dfb9f5566e941d7a93baab9695ec22e/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/3cb246ac67ca0826ca86f8adb5c5c1b8802c4f96ca330456aea67aec02231f9c/shm
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/56cbc698b06f57937128eadc74cc098c4dfb9f5566e941d7a93baab9695ec22e/rootfs
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/3cb246ac67ca0826ca86f8adb5c5c1b8802c4f96ca330456aea67aec02231f9c/rootfs
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/f3e2bedafb61411951557c6e66d037536240bf25a185e6b3e6da0b6ad0b91a38/rootfs
tmpfs 378M 0 378M 0% /run/user/0
[root@alva-anstawx03 ~]#
其他节点也一样：
[root@alva-anstawx02 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 68M 1.8G 4% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/vg_system-lv_root 9.8G 2.1G 7.2G 23% /
/dev/sda1 976M 206M 704M 23% /boot
/dev/mapper/vg_system-lv_opt 3.9G 72M 3.6G 2% /opt
/dev/mapper/vg_system-lv_var 19G 1.1G 17G 6% /var
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/31e01070-282a-453b-8e7f-fe0d93e359ec/volumes/kubernetes.io~projected/kube-api-access-4qhqs
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/4208e857-28e7-4005-bbe1-8bed0b08548b/volumes/kubernetes.io~projected/kube-api-access-bvjhg
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/f3b43b5c1e46236e3e01536cff2089c788e0b39e34e43165608dbb2ea9906cb5/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/285018acde46e519f9ae74551d06028304ca19ab76813ed1ca43a4b6e617e4f4/shm
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/285018acde46e519f9ae74551d06028304ca19ab76813ed1ca43a4b6e617e4f4/rootfs
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/f3b43b5c1e46236e3e01536cff2089c788e0b39e34e43165608dbb2ea9906cb5/rootfs
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/42aaae5f8c681ffa5fd0bf6ed3fcf4d8447962131459d4592d1bbb73a320edca/rootfs
tmpfs 378M 0 378M 0% /run/user/0
[root@alva-anstawx02 ~]#
以下是 kubectl 的输出：
[root@alva-anstawx01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
alva-anstawx01.<Domain Name> Ready control-plane,master 8d v1.21.1
alva-anstawx02.<Domain Name> NotReady <none> 43h v1.21.1
alva-anstawx03.<Domain Name> NotReady <none> 43h v1.21.1
[root@alva-anstawx01 ~]# kubectl describe pod calico-node-dshv9 -n kube-system
Name: calico-node-dshv9
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Node: alva-anstawx03.<Domain Name>/10.4.9.96
Start Time: Wed, 09 Jun 2021 20:39:52 +0200
Labels: controller-revision-hash=c54f47b5c
k8s-app=calico-node
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 10.4.9.96
IPs:
IP: 10.4.9.96
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID:
Image: docker.io/calico/cni:v3.19.1
Image ID:
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
install-cni:
Container ID:
Image: docker.io/calico/cni:v3.19.1
Image ID:
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
flexvol-driver:
Container ID:
Image: docker.io/calico/pod2daemon-flexvol:v3.19.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/host/driver from flexvol-driver-host (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
Containers:
calico-node:
Container ID:
Image: docker.io/calico/node:v3.19.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_HEALTHENABLED: true
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/ from sysfs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sysfs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
flexvol-driver-host:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
HostPathType: DirectoryOrCreate
kube-api-access-9bh4m:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13h default-scheduler Successfully assigned kube-system/calico-node-dshv9 to alva-anstawx03.<Domain Name>
Warning Failed 13h (x2 over 13h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 13h (x4 over 13h) kubelet Pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 13h (x4 over 13h) kubelet Error: ErrImagePull
Warning Failed 13h (x2 over 13h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 13h (x6 over 13h) kubelet Error: ImagePullBackOff
Normal BackOff 12h (x61 over 13h) kubelet Back-off pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x2 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 12h (x4 over 12h) kubelet Pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x4 over 12h) kubelet Error: ErrImagePull
Warning Failed 12h (x2 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 12h (x6 over 12h) kubelet Error: ImagePullBackOff
Normal BackOff 12h (x18 over 12h) kubelet Back-off pulling image "docker.io/calico/cni:v3.19.1"
Normal Pulling 12h (x4 over 12h) kubelet Pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x4 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 12h (x4 over 12h) kubelet Error: ErrImagePull
Warning Failed 12h (x6 over 12h) kubelet Error: ImagePullBackOff
Normal BackOff 12h (x81 over 12h) kubelet Back-off pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x2 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 12h (x4 over 12h) kubelet Pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x2 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 12h (x4 over 12h) kubelet Error: ErrImagePull
Warning Failed 12h (x6 over 12h) kubelet Error: ImagePullBackOff
Normal BackOff 4m45s (x3192 over 12h) kubelet Back-off pulling image "docker.io/calico/cni:v3.19.1"
[root@alva-anstawx01 ~]# kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-78d6f96c7b-wb96g 1/1 Running 1 13h 192.168.228.198 alva-anstawx01.<Domain Name> <none> <none>
kube-system calico-node-dshv9 0/1 Init:ImagePullBackOff 0 13h 10.4.9.96 alva-anstawx03.<Domain Name> <none> <none>
kube-system calico-node-rfrnq 0/1 Init:ImagePullBackOff 0 13h 10.4.9.80 alva-anstawx02.<Domain Name> <none> <none>
kube-system calico-node-sl864 1/1 Running 1 13h 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system coredns-558bd4d5db-c8p97 1/1 Running 2 8d 192.168.228.200 alva-anstawx01.<Domain Name> <none> <none>
kube-system coredns-558bd4d5db-hjb8g 1/1 Running 2 8d 192.168.228.199 alva-anstawx01.<Domain Name> <none> <none>
kube-system etcd-alva-anstawx01.<Domain Name> 1/1 Running 4 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system kube-apiserver-alva-anstawx01.<Domain Name> 1/1 Running 4 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system kube-controller-manager-alva-anstawx01.<Domain Name> 1/1 Running 4 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system kube-proxy-b8ft2 1/1 Running 2 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system kube-proxy-frr7c 1/1 Running 4 43h 10.4.9.80 alva-anstawx02.<Domain Name> <none> <none>
kube-system kube-proxy-ztxbf 1/1 Running 4 43h 10.4.9.96 alva-anstawx03.<Domain Name> <none> <none>
kube-system kube-scheduler-alva-anstawx01.<Domain Name> 1/1 Running 4 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
看到整个日志行条目后
Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
我们对这部分感兴趣：
429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit.
看来问题不在于 calico/kubernetes 集群，而在于为其拉取 docker 映像。
所述
The rate limits of 100 container image requests per six hours for
anonymous usage, and 200 container image requests per six hours for
free Docker accounts are now in effect. Image requests exceeding these
limits will be denied until the six hour window elapses.
如果没有从该 IP 到 docker 集线器的拉取请求，它现在应该下载图像。为此，您可以删除负责节点上 calico 网络的 pods，因为它是一个部署，调度程序将为您重新创建 pods。为此，运行:
kubectl delete pod calico-node-rfrnq -n kube-system
kubectl delete pod calico-node-dshv9 -n kube-system
如果出于任何原因您的节点仍然无法从 docker 集线器中提取 docker 图像，则您在控制平面上有 calico pod 运行ning。这意味着您可以手动将 docker 图像从控制平面传输到工作节点。
Docker
在控制平面上 运行 以下命令：
docker save -o ~/calico-cni.tar calico/cni:v3.19.1
使用 sftp、scp 或任何其他解决方案将您获得的文件复制到工作节点。
在工作节点上 运行:
docker load -i calico-cni.tar
如果创建映像没有开始，请考虑删除工作节点上的 pods 以强制 kube 调度程序重新安排要创建的这些 pods。
请注意，如果您采用手动方式，可能还需要下载其他图片。例如。在我的工作节点上有四个与 calico 相关的图像：
docker images | grep calico
calico/node v3.19.1 c4d75af7e098 3 weeks ago 168MB
calico/pod2daemon-flexvol v3.19.1 5660150975fb 3 weeks ago 21.7MB
calico/cni v3.19.1 5749e8b276f9 3 weeks ago 146MB
calico/kube-controllers v3.19.1 5d3d5ddc8605 3 weeks ago 60.6MB
容器D
使用 ContainerD 导出镜像
ctr image export <output-filename> <image-name>
示例：
ctr image export calico-node-v3.11.2.tar \
docker.io/calico/node:v3.11.2
将您获得的文件复制到工作节点。
ctr image import <filename-from-previous-step>
请查找 ContainerD
的语法
我遇到了同样的问题，但与 blew msg 有点不同：
“在 /etc/cni/net.d 中找不到网络”
我通过以下操作修复了它：
- 创建文件夹 /etc/cni/net.d
- 复制/etc/cni/net.d/10-flannel.conflist到当前故障节点
- systemctl 重启 kubelet
我安装了三个节点的 Kubernetes 集群，控制节点看起来没问题，当我尝试加入其他两个节点时，两个节点的状态都是：未就绪
在控制节点上：
[root@alva-anstawx01 /]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
alva-anstawx01.<domain name> Ready control-plane,master 7d20h v1.21.1
alva-anstawx02.<domain name> NotReady <none> 22h v1.21.1
alva-anstawx03.<domain name> NotReady <none> 22h v1.21.1
PODs 看起来还不错 运行:
NAME READY STATUS RESTARTS AGE
coredns-558bd4d5db-c8p97 1/1 Running 0 7d20h
coredns-558bd4d5db-hjb8g 1/1 Running 0 7d20h
etcd-alva-anstawx01.alv.autoliv.int 1/1 Running 2 7d20h
kube-apiserver-alva-anstawx01.alv.autoliv.int 1/1 Running 2 7d20h
kube-controller-manager-alva-anstawx01.alv.autoliv.int 1/1 Running 2 7d20h
kube-proxy-b8ft2 1/1 Running 0 7d20h
kube-proxy-frr7c 1/1 Running 0 23h
kube-proxy-ztxbf 1/1 Running 0 23h
kube-scheduler-alva-anstawx01.alv.autoliv.int 1/1 Running 2 7d20h
进一步检查日志，发现缺少某些内容，因此 CNI 插件在这些节点上启动，但不确定如何继续：
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 09 Jun 2021 12:24:34 +0200 Tue, 08 Jun 2021 14:00:45 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 09 Jun 2021 12:24:34 +0200 Tue, 08 Jun 2021 14:00:45 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 09 Jun 2021 12:24:34 +0200 Tue, 08 Jun 2021 14:00:45 +0200 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Wed, 09 Jun 2021 12:24:34 +0200 Tue, 08 Jun 2021 14:00:45 +0200 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
我在每个节点上都有一个网络接口
在受影响的节点上：
Jun 09 12:34:19 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:19.209657 1026 kubelet.go:2211] "Container runtime network not ready" networkReady="N
Jun 09 12:34:19 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:19.698034 1026 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"Sta
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817375 1026 remote_image.go:114] "PullImage from image service failed" err="rpc er
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817429 1026 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code =
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817627 1026 kuberuntime_manager.go:864] container &Container{Name:calico-typha,Ima
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817706 1026 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"Sta
Jun 09 12:34:24 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:24.211195 1026 kubelet.go:2211] "Container runtime network not ready" networkReady="N
我使用了 Calico 默认配置并且我在每个节点上都有一个接口：
控制节点：
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:ac:64:8c brd ff:ff:ff:ff:ff:ff
inet 10.4.9.73/21 brd 10.4.15.255 scope global ens192
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feac:648c/64 scope link
valid_lft forever preferred_lft forever
3: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 66:c5:7d:06:e5:fe brd ff:ff:ff:ff:ff:ff
inet 192.168.228.192/32 scope global vxlan.calico
valid_lft forever preferred_lft forever
inet6 fe80::64c5:7dff:fe06:e5fe/64 scope link
valid_lft forever preferred_lft forever
4: cali5441eeb56bd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
5: cali389c5f98ecc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
6: calicc306a285eb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
其他节点：
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:ac:67:61 brd ff:ff:ff:ff:ff:ff
inet 10.4.9.80/21 brd 10.4.15.255 scope global ens192
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feac:6761/64 scope link
valid_lft forever preferred_lft forever
谁能帮我解决如何在其他两个节点上初始化 calico 的问题？
编辑：
我已经解决了 /var space 的问题，但没有帮助：
[root@alva-anstawx03 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 60M 1.8G 4% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/vg_system-lv_root 9.8G 2.1G 7.2G 23% /
/dev/sda1 976M 206M 704M 23% /boot
/dev/mapper/vg_system-lv_var 19G 1.1G 17G 6% /var
/dev/mapper/vg_system-lv_opt 3.9G 72M 3.6G 2% /opt
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/77b1f7f0-8598-4640-af2f-e960c4c76120/volumes/kubernetes.io~projected/kube-api-access-7xnp8
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/4398eeeb-0f74-477c-a066-403ecab4abe1/volumes/kubernetes.io~projected/kube-api-access-9bh4m
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/56cbc698b06f57937128eadc74cc098c4dfb9f5566e941d7a93baab9695ec22e/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/3cb246ac67ca0826ca86f8adb5c5c1b8802c4f96ca330456aea67aec02231f9c/shm
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/56cbc698b06f57937128eadc74cc098c4dfb9f5566e941d7a93baab9695ec22e/rootfs
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/3cb246ac67ca0826ca86f8adb5c5c1b8802c4f96ca330456aea67aec02231f9c/rootfs
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/f3e2bedafb61411951557c6e66d037536240bf25a185e6b3e6da0b6ad0b91a38/rootfs
tmpfs 378M 0 378M 0% /run/user/0
[root@alva-anstawx03 ~]#
其他节点也一样：
[root@alva-anstawx02 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 68M 1.8G 4% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/vg_system-lv_root 9.8G 2.1G 7.2G 23% /
/dev/sda1 976M 206M 704M 23% /boot
/dev/mapper/vg_system-lv_opt 3.9G 72M 3.6G 2% /opt
/dev/mapper/vg_system-lv_var 19G 1.1G 17G 6% /var
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/31e01070-282a-453b-8e7f-fe0d93e359ec/volumes/kubernetes.io~projected/kube-api-access-4qhqs
tmpfs 1.9G 12K 1.9G 1% /var/lib/kubelet/pods/4208e857-28e7-4005-bbe1-8bed0b08548b/volumes/kubernetes.io~projected/kube-api-access-bvjhg
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/f3b43b5c1e46236e3e01536cff2089c788e0b39e34e43165608dbb2ea9906cb5/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/285018acde46e519f9ae74551d06028304ca19ab76813ed1ca43a4b6e617e4f4/shm
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/285018acde46e519f9ae74551d06028304ca19ab76813ed1ca43a4b6e617e4f4/rootfs
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/f3b43b5c1e46236e3e01536cff2089c788e0b39e34e43165608dbb2ea9906cb5/rootfs
overlay 19G 1.1G 17G 6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/42aaae5f8c681ffa5fd0bf6ed3fcf4d8447962131459d4592d1bbb73a320edca/rootfs
tmpfs 378M 0 378M 0% /run/user/0
[root@alva-anstawx02 ~]#
以下是 kubectl 的输出：
[root@alva-anstawx01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
alva-anstawx01.<Domain Name> Ready control-plane,master 8d v1.21.1
alva-anstawx02.<Domain Name> NotReady <none> 43h v1.21.1
alva-anstawx03.<Domain Name> NotReady <none> 43h v1.21.1
[root@alva-anstawx01 ~]# kubectl describe pod calico-node-dshv9 -n kube-system
Name: calico-node-dshv9
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Node: alva-anstawx03.<Domain Name>/10.4.9.96
Start Time: Wed, 09 Jun 2021 20:39:52 +0200
Labels: controller-revision-hash=c54f47b5c
k8s-app=calico-node
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 10.4.9.96
IPs:
IP: 10.4.9.96
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID:
Image: docker.io/calico/cni:v3.19.1
Image ID:
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
install-cni:
Container ID:
Image: docker.io/calico/cni:v3.19.1
Image ID:
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
flexvol-driver:
Container ID:
Image: docker.io/calico/pod2daemon-flexvol:v3.19.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/host/driver from flexvol-driver-host (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
Containers:
calico-node:
Container ID:
Image: docker.io/calico/node:v3.19.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_HEALTHENABLED: true
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/ from sysfs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sysfs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
flexvol-driver-host:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
HostPathType: DirectoryOrCreate
kube-api-access-9bh4m:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13h default-scheduler Successfully assigned kube-system/calico-node-dshv9 to alva-anstawx03.<Domain Name>
Warning Failed 13h (x2 over 13h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 13h (x4 over 13h) kubelet Pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 13h (x4 over 13h) kubelet Error: ErrImagePull
Warning Failed 13h (x2 over 13h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 13h (x6 over 13h) kubelet Error: ImagePullBackOff
Normal BackOff 12h (x61 over 13h) kubelet Back-off pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x2 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 12h (x4 over 12h) kubelet Pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x4 over 12h) kubelet Error: ErrImagePull
Warning Failed 12h (x2 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 12h (x6 over 12h) kubelet Error: ImagePullBackOff
Normal BackOff 12h (x18 over 12h) kubelet Back-off pulling image "docker.io/calico/cni:v3.19.1"
Normal Pulling 12h (x4 over 12h) kubelet Pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x4 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 12h (x4 over 12h) kubelet Error: ErrImagePull
Warning Failed 12h (x6 over 12h) kubelet Error: ImagePullBackOff
Normal BackOff 12h (x81 over 12h) kubelet Back-off pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x2 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 12h (x4 over 12h) kubelet Pulling image "docker.io/calico/cni:v3.19.1"
Warning Failed 12h (x2 over 12h) kubelet Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 12h (x4 over 12h) kubelet Error: ErrImagePull
Warning Failed 12h (x6 over 12h) kubelet Error: ImagePullBackOff
Normal BackOff 4m45s (x3192 over 12h) kubelet Back-off pulling image "docker.io/calico/cni:v3.19.1"
[root@alva-anstawx01 ~]# kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-78d6f96c7b-wb96g 1/1 Running 1 13h 192.168.228.198 alva-anstawx01.<Domain Name> <none> <none>
kube-system calico-node-dshv9 0/1 Init:ImagePullBackOff 0 13h 10.4.9.96 alva-anstawx03.<Domain Name> <none> <none>
kube-system calico-node-rfrnq 0/1 Init:ImagePullBackOff 0 13h 10.4.9.80 alva-anstawx02.<Domain Name> <none> <none>
kube-system calico-node-sl864 1/1 Running 1 13h 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system coredns-558bd4d5db-c8p97 1/1 Running 2 8d 192.168.228.200 alva-anstawx01.<Domain Name> <none> <none>
kube-system coredns-558bd4d5db-hjb8g 1/1 Running 2 8d 192.168.228.199 alva-anstawx01.<Domain Name> <none> <none>
kube-system etcd-alva-anstawx01.<Domain Name> 1/1 Running 4 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system kube-apiserver-alva-anstawx01.<Domain Name> 1/1 Running 4 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system kube-controller-manager-alva-anstawx01.<Domain Name> 1/1 Running 4 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system kube-proxy-b8ft2 1/1 Running 2 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
kube-system kube-proxy-frr7c 1/1 Running 4 43h 10.4.9.80 alva-anstawx02.<Domain Name> <none> <none>
kube-system kube-proxy-ztxbf 1/1 Running 4 43h 10.4.9.96 alva-anstawx03.<Domain Name> <none> <none>
kube-system kube-scheduler-alva-anstawx01.<Domain Name> 1/1 Running 4 8d 10.4.9.73 alva-anstawx01.<Domain Name> <none> <none>
看到整个日志行条目后
Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
我们对这部分感兴趣：
429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit.
看来问题不在于 calico/kubernetes 集群，而在于为其拉取 docker 映像。
所述The rate limits of 100 container image requests per six hours for anonymous usage, and 200 container image requests per six hours for free Docker accounts are now in effect. Image requests exceeding these limits will be denied until the six hour window elapses.
如果没有从该 IP 到 docker 集线器的拉取请求，它现在应该下载图像。为此，您可以删除负责节点上 calico 网络的 pods，因为它是一个部署，调度程序将为您重新创建 pods。为此，运行:
kubectl delete pod calico-node-rfrnq -n kube-system
kubectl delete pod calico-node-dshv9 -n kube-system
如果出于任何原因您的节点仍然无法从 docker 集线器中提取 docker 图像，则您在控制平面上有 calico pod 运行ning。这意味着您可以手动将 docker 图像从控制平面传输到工作节点。
Docker
在控制平面上 运行 以下命令：
docker save -o ~/calico-cni.tar calico/cni:v3.19.1
使用 sftp、scp 或任何其他解决方案将您获得的文件复制到工作节点。
在工作节点上 运行:
docker load -i calico-cni.tar
如果创建映像没有开始，请考虑删除工作节点上的 pods 以强制 kube 调度程序重新安排要创建的这些 pods。
请注意，如果您采用手动方式，可能还需要下载其他图片。例如。在我的工作节点上有四个与 calico 相关的图像：
docker images | grep calico
calico/node v3.19.1 c4d75af7e098 3 weeks ago 168MB
calico/pod2daemon-flexvol v3.19.1 5660150975fb 3 weeks ago 21.7MB
calico/cni v3.19.1 5749e8b276f9 3 weeks ago 146MB
calico/kube-controllers v3.19.1 5d3d5ddc8605 3 weeks ago 60.6MB
容器D
使用 ContainerD 导出镜像
ctr image export <output-filename> <image-name>
示例：
ctr image export calico-node-v3.11.2.tar \
docker.io/calico/node:v3.11.2
将您获得的文件复制到工作节点。
ctr image import <filename-from-previous-step>
请查找 ContainerD
的语法我遇到了同样的问题，但与 blew msg 有点不同： “在 /etc/cni/net.d 中找不到网络”
我通过以下操作修复了它：
- 创建文件夹 /etc/cni/net.d
- 复制/etc/cni/net.d/10-flannel.conflist到当前故障节点
- systemctl 重启 kubelet