就绪探测失败：获取 http://10.244.0.3:8181/ready：拨打 tcp 10.244.0.3:8181：连接：连接被拒绝
Readiness probe failed: Get http://10.244.0.3:8181/ready: dial tcp 10.244.0.3:8181: connect: connection refused
我是 Kubernetes 的新手，试图通过 kubeadm 构建 kubernetes 集群，我已经跨过这个阶段直到 kube init（使用 args PFB 命令 FYR）但是 CoreDNS pod 失败 CrashLoopBackOff 状态。我试图检查该 pod 的事件并收到以下错误。我不确定它调用的是哪个服务以及为什么连接被拒绝。你能帮我解决这个问题吗？我执行过的PFB命令。
kubeadm 初始化：
kubeadm init --pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=192.168.56.2
CNI网络插件：
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
命名空间命令：
root@kube-master:/etc# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-5z9f6 0/1 CrashLoopBackOff 32 143m
kube-system coredns-66bff467f8-tsjdj 0/1 CrashLoopBackOff 32 143m
kube-system etcd-kube-master 1/1 Running 0 143m
kube-system kube-apiserver-kube-master 1/1 Running 0 143m
kube-system kube-controller-manager-kube-master 1/1 Running 0 143m
kube-system kube-flannel-ds-amd64-2kklk 1/1 Running 0 123m
kube-system kube-proxy-5p9jf 1/1 Running 0 143m
kube-system kube-scheduler-kube-master 1/1 Running 0 143m
coredns 事件检查：
root@kube-master:/etc# kubectl describe pod coredns-66bff467f8-tsjdj -n kube-system
Name: coredns-66bff467f8-tsjdj
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: kube-master/192.168.0.142
Start Time: Fri, 10 Jul 2020 15:54:45 -0400
Labels: k8s-app=kube-dns
pod-template-hash=66bff467f8
Annotations: <none>
Status: Running
IP: 10.244.0.3
IPs:
IP: 10.244.0.3
Controlled By: ReplicaSet/coredns-66bff467f8
Containers:
coredns:
Container ID: docker://20d8697b21f1bfe955de6769abfbb18c658f3c4134a2505f580b8146c998871d
Image: k8s.gcr.io/coredns:1.6.7
Image ID: docker-pullable://k8s.gcr.io/coredns@sha256:2c8d61c46f484d881db43b34d13ca47a269336e576c81cf007ca740fa9ec0800
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 10 Jul 2020 17:55:00 -0400
Finished: Fri, 10 Jul 2020 17:55:01 -0400
Ready: False
Restart Count: 32
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-2w9w6 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-2w9w6:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-2w9w6
Optional: false
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 13m (x523 over 123m) kubelet, kube-master Back-off restarting failed container
Warning Unhealthy 9m18s kubelet, kube-master Readiness probe failed: Get http://10.244.0.3:8181/ready: dial tcp 10.244.0.3:8181: connect: connection refused
Normal Pulled 7m42s (x4 over 9m23s) kubelet, kube-master Container image "k8s.gcr.io/coredns:1.6.7" already present on machine
Normal Created 7m42s (x4 over 9m22s) kubelet, kube-master Created container coredns
Normal Started 7m41s (x4 over 9m21s) kubelet, kube-master Started container coredns
Warning BackOff 4m14s (x26 over 9m17s) kubelet, kube-master Back-off restarting failed container
你能帮我解决这个问题吗？
谢谢
您必须编辑您的 coredns 配置映射并添加就绪标志：
kubectl -n kube-system edit coredns -o yaml
您将获得编辑的 YAML 定义。添加行 ready :8181 如下（如果不存在）：
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.as-gmbh.de in-addr.arpa ip6.arpa {
pods verified
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
ready :8181 # <---- Ignore other lines and Focus Here
forward . 172.18.2.21
cache 30
loop
reload
loadbalance
}
我在旧版本和最新版本中都发现了这个问题。 coredns 正向查询处于循环状态，一旦检测到循环就会退出。请查看此 URL 了解更多详情 - https://coredns.io/plugins/loop/#troubleshooting
我确实检查了我的 /etc/resolv.conf，我的名称服务器是 127.0.1.1，因此它检测到循环以转发 DNS 查询。所以我已经按照下面更改了我的网络配置
https://askubuntu.com/questions/627899/nameserver-127-0-1-1-in-resolv-conf-wont-go-away
现在工作正常。即使我没有编辑 coredns yaml 文件来添加 8181 端口以准备就绪。
root@kube-master:/etc# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-522nr 1/1 Running 0 11m
kube-system coredns-66bff467f8-8qskt 1/1 Running 0 11m
kube-system etcd-kube-master 1/1 Running 0 11m
kube-system kube-apiserver-kube-master 1/1 Running 0 11m
kube-system kube-controller-manager-kube-master 1/1 Running 0 11m
kube-system kube-flannel-ds-amd64-4jbpb 1/1 Running 0 10m
kube-system kube-proxy-hz94b 1/1 Running 0 11m
kube-system kube-scheduler-kube-master 1/1 Running 0 11m
coredns 日志
root@kube-master:/etc# kubectl logs coredns-66bff467f8-522nr -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
coredns pod 事件：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 118s (x12 over 3m27s) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
Normal Scheduled 107s default-scheduler Successfully assigned kube-system/coredns-66bff467f8-522nr to kube-master
Normal Pulled 98s kubelet, kube-master Container image "k8s.gcr.io/coredns:1.6.7" already present on machine
Normal Created 96s kubelet, kube-master Created container coredns
Normal Started 94s kubelet, kube-master Started container coredns
感谢@Abdennour TOUMI 和@HelloWorld 的帮助
我是 Kubernetes 的新手，试图通过 kubeadm 构建 kubernetes 集群，我已经跨过这个阶段直到 kube init（使用 args PFB 命令 FYR）但是 CoreDNS pod 失败 CrashLoopBackOff 状态。我试图检查该 pod 的事件并收到以下错误。我不确定它调用的是哪个服务以及为什么连接被拒绝。你能帮我解决这个问题吗？我执行过的PFB命令。
kubeadm 初始化：
kubeadm init --pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=192.168.56.2
CNI网络插件：
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
命名空间命令：
root@kube-master:/etc# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-5z9f6 0/1 CrashLoopBackOff 32 143m
kube-system coredns-66bff467f8-tsjdj 0/1 CrashLoopBackOff 32 143m
kube-system etcd-kube-master 1/1 Running 0 143m
kube-system kube-apiserver-kube-master 1/1 Running 0 143m
kube-system kube-controller-manager-kube-master 1/1 Running 0 143m
kube-system kube-flannel-ds-amd64-2kklk 1/1 Running 0 123m
kube-system kube-proxy-5p9jf 1/1 Running 0 143m
kube-system kube-scheduler-kube-master 1/1 Running 0 143m
coredns 事件检查：
root@kube-master:/etc# kubectl describe pod coredns-66bff467f8-tsjdj -n kube-system
Name: coredns-66bff467f8-tsjdj
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: kube-master/192.168.0.142
Start Time: Fri, 10 Jul 2020 15:54:45 -0400
Labels: k8s-app=kube-dns
pod-template-hash=66bff467f8
Annotations: <none>
Status: Running
IP: 10.244.0.3
IPs:
IP: 10.244.0.3
Controlled By: ReplicaSet/coredns-66bff467f8
Containers:
coredns:
Container ID: docker://20d8697b21f1bfe955de6769abfbb18c658f3c4134a2505f580b8146c998871d
Image: k8s.gcr.io/coredns:1.6.7
Image ID: docker-pullable://k8s.gcr.io/coredns@sha256:2c8d61c46f484d881db43b34d13ca47a269336e576c81cf007ca740fa9ec0800
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 10 Jul 2020 17:55:00 -0400
Finished: Fri, 10 Jul 2020 17:55:01 -0400
Ready: False
Restart Count: 32
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-2w9w6 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-2w9w6:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-2w9w6
Optional: false
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 13m (x523 over 123m) kubelet, kube-master Back-off restarting failed container
Warning Unhealthy 9m18s kubelet, kube-master Readiness probe failed: Get http://10.244.0.3:8181/ready: dial tcp 10.244.0.3:8181: connect: connection refused
Normal Pulled 7m42s (x4 over 9m23s) kubelet, kube-master Container image "k8s.gcr.io/coredns:1.6.7" already present on machine
Normal Created 7m42s (x4 over 9m22s) kubelet, kube-master Created container coredns
Normal Started 7m41s (x4 over 9m21s) kubelet, kube-master Started container coredns
Warning BackOff 4m14s (x26 over 9m17s) kubelet, kube-master Back-off restarting failed container
你能帮我解决这个问题吗？
谢谢
您必须编辑您的 coredns 配置映射并添加就绪标志：
kubectl -n kube-system edit coredns -o yaml
您将获得编辑的 YAML 定义。添加行 ready :8181 如下（如果不存在）：
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.as-gmbh.de in-addr.arpa ip6.arpa {
pods verified
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
ready :8181 # <---- Ignore other lines and Focus Here
forward . 172.18.2.21
cache 30
loop
reload
loadbalance
}
我在旧版本和最新版本中都发现了这个问题。 coredns 正向查询处于循环状态，一旦检测到循环就会退出。请查看此 URL 了解更多详情 - https://coredns.io/plugins/loop/#troubleshooting
我确实检查了我的 /etc/resolv.conf，我的名称服务器是 127.0.1.1，因此它检测到循环以转发 DNS 查询。所以我已经按照下面更改了我的网络配置
https://askubuntu.com/questions/627899/nameserver-127-0-1-1-in-resolv-conf-wont-go-away
现在工作正常。即使我没有编辑 coredns yaml 文件来添加 8181 端口以准备就绪。
root@kube-master:/etc# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-522nr 1/1 Running 0 11m
kube-system coredns-66bff467f8-8qskt 1/1 Running 0 11m
kube-system etcd-kube-master 1/1 Running 0 11m
kube-system kube-apiserver-kube-master 1/1 Running 0 11m
kube-system kube-controller-manager-kube-master 1/1 Running 0 11m
kube-system kube-flannel-ds-amd64-4jbpb 1/1 Running 0 10m
kube-system kube-proxy-hz94b 1/1 Running 0 11m
kube-system kube-scheduler-kube-master 1/1 Running 0 11m
coredns 日志
root@kube-master:/etc# kubectl logs coredns-66bff467f8-522nr -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
coredns pod 事件：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 118s (x12 over 3m27s) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
Normal Scheduled 107s default-scheduler Successfully assigned kube-system/coredns-66bff467f8-522nr to kube-master
Normal Pulled 98s kubelet, kube-master Container image "k8s.gcr.io/coredns:1.6.7" already present on machine
Normal Created 96s kubelet, kube-master Created container coredns
Normal Started 94s kubelet, kube-master Started container coredns
感谢@Abdennour TOUMI 和@HelloWorld 的帮助