即使资源可用，Pod 也未被安排
Pod not being scheduled even though resources available
我是 运行 GCP 中的一个 4 工作节点集群。
我节点的当前状态是：
节点A
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system calico-node-bfpbd 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-7br2g 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-7kvcm 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
kube-system tiller-deploy-56c4cf647b-5vsvb 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 310m (7%) 620m (15%)
memory 174Mi (1%) 1006Mi (7%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:08 -0400 Thu, 01 Nov 2018 21:31:08 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:31:06 -0400 KubeletReady kubelet is posting ready status
节点 B
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
jenkins-test jenkins-master 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system calico-node-qglbv 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-g74ff 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-bvczb 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 310m (7%) 620m (15%)
memory 174Mi (1%) 1006Mi (7%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:06 -0400 Thu, 01 Nov 2018 21:31:06 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:31:06 -0400 KubeletReady kubelet is posting ready status
节点 C
Non-terminated Pods: (3 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system calico-node-w9px6 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-4r2ck 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-r92xs 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:01 -0400 Thu, 01 Nov 2018 21:31:01 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:31:09 -0400 KubeletReady kubelet is posting ready status
节点 D
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system addons-kubernetes-dashboard-8656b6fc5f-68wzm 50m (1%) 200m (5%) 50Mi (0%) 256Mi (1%)
kube-system addons-nginx-ingress-controller-77579b6d64-sqzl7 100m (2%) 300m (7%) 100Mi (0%) 512Mi (3%)
kube-system addons-nginx-ingress-nginx-ingress-k8s-backend-5d6d4598ff-nfzt4 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system calico-node-h2t5b 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system coredns-5c554d9f6f-fnwqq 100m (2%) 200m (5%) 15Mi (0%) 80Mi (0%)
kube-system kube-proxy-bfhjr 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system metrics-server-7f4cbf557d-985sj 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system node-exporter-frdrd 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
kube-system vpn-shoot-7bcd5f4bb-88sc7 100m (2%) 300m (7%) 128Mi (0%) 512Mi (3%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:30:54 -0400 Thu, 01 Nov 2018 21:30:54 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:31:02 -0400 KubeletReady kubelet is posting ready status
如所有结果所示，我的节点都很健康并且有充足的资源，每个节点有 4 个 CPU，每个节点有 16GB 内存。
现在，当我尝试在命名空间中部署第二个 statefulSet 时，Pod 仍处于 Pending 状态。描述显示以下消息：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m (x123 over 7m) default-scheduler pod has unbound PersistentVolumeClaims (repeated 2 times)
Normal NotTriggerScaleUp 12s (x26 over 6m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added)
我还耗尽了 2 个节点，标记了这两个节点并将 nodeSelector 附加到我的 statefulset 以仅部署在这 2 个几乎为空的节点上，但结果是相同的。
我不确定为什么我的 pod 正在尝试 scale-up。这不是意图。任何帮助将不胜感激。
kubectl get po -n jenkins-test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
jenkins-agent 0/1 Pending 0 12m <none> <none> <none>
jenkins-master 1/1 Running 0 22m 100.96.1.2 shoot--t--csp-worker-hqh6g-z1-6df8f7dc66-bcj6t <none>
pod has unbound PersistentVolumeClaims ，我觉得是这个问题的关键。关于scale up，Pod是通过'scale from 0 to any instance you defined'创建的。
我是 运行 GCP 中的一个 4 工作节点集群。
我节点的当前状态是：
节点A
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system calico-node-bfpbd 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-7br2g 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-7kvcm 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
kube-system tiller-deploy-56c4cf647b-5vsvb 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 310m (7%) 620m (15%)
memory 174Mi (1%) 1006Mi (7%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:08 -0400 Thu, 01 Nov 2018 21:31:08 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:50 -0400 Thu, 01 Nov 2018 21:31:06 -0400 KubeletReady kubelet is posting ready status
节点 B
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
jenkins-test jenkins-master 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system calico-node-qglbv 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-g74ff 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-bvczb 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 310m (7%) 620m (15%)
memory 174Mi (1%) 1006Mi (7%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:06 -0400 Thu, 01 Nov 2018 21:31:06 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:30:46 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:49 -0400 Thu, 01 Nov 2018 21:31:06 -0400 KubeletReady kubelet is posting ready status
节点 C
Non-terminated Pods: (3 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system calico-node-w9px6 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system kube-proxy-4r2ck 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system node-exporter-r92xs 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:31:01 -0400 Thu, 01 Nov 2018 21:31:01 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:30:49 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:42 -0400 Thu, 01 Nov 2018 21:31:09 -0400 KubeletReady kubelet is posting ready status
节点 D
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system addons-kubernetes-dashboard-8656b6fc5f-68wzm 50m (1%) 200m (5%) 50Mi (0%) 256Mi (1%)
kube-system addons-nginx-ingress-controller-77579b6d64-sqzl7 100m (2%) 300m (7%) 100Mi (0%) 512Mi (3%)
kube-system addons-nginx-ingress-nginx-ingress-k8s-backend-5d6d4598ff-nfzt4 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system calico-node-h2t5b 250m (6%) 500m (12%) 100Mi (0%) 700Mi (5%)
kube-system coredns-5c554d9f6f-fnwqq 100m (2%) 200m (5%) 15Mi (0%) 80Mi (0%)
kube-system kube-proxy-bfhjr 50m (1%) 100m (2%) 64Mi (0%) 256Mi (1%)
kube-system metrics-server-7f4cbf557d-985sj 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system node-exporter-frdrd 10m (0%) 20m (0%) 10Mi (0%) 50Mi (0%)
kube-system vpn-shoot-7bcd5f4bb-88sc7 100m (2%) 300m (7%) 128Mi (0%) 512Mi (3%)
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 01 Nov 2018 21:30:54 -0400 Thu, 01 Nov 2018 21:30:54 -0400 RouteCreated RouteController created a route
OutOfDisk False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:30:42 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 01 Nov 2018 21:48:45 -0400 Thu, 01 Nov 2018 21:31:02 -0400 KubeletReady kubelet is posting ready status
如所有结果所示，我的节点都很健康并且有充足的资源，每个节点有 4 个 CPU，每个节点有 16GB 内存。
现在，当我尝试在命名空间中部署第二个 statefulSet 时，Pod 仍处于 Pending 状态。描述显示以下消息：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m (x123 over 7m) default-scheduler pod has unbound PersistentVolumeClaims (repeated 2 times)
Normal NotTriggerScaleUp 12s (x26 over 6m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added)
我还耗尽了 2 个节点，标记了这两个节点并将 nodeSelector 附加到我的 statefulset 以仅部署在这 2 个几乎为空的节点上，但结果是相同的。
我不确定为什么我的 pod 正在尝试 scale-up。这不是意图。任何帮助将不胜感激。
kubectl get po -n jenkins-test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
jenkins-agent 0/1 Pending 0 12m <none> <none> <none>
jenkins-master 1/1 Running 0 22m 100.96.1.2 shoot--t--csp-worker-hqh6g-z1-6df8f7dc66-bcj6t <none>
pod has unbound PersistentVolumeClaims ，我觉得是这个问题的关键。关于scale up，Pod是通过'scale from 0 to any instance you defined'创建的。