即使资源可用,Pod 也未被安排

Pod not being scheduled even though resources available

我是 运行 GCP 中的一个 4 工作节点集群。

我节点的当前状态是:

节点A

Non-terminated Pods:         (4 in total)
  Namespace                  Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                              ------------  ----------  ---------------  -------------
  kube-system                calico-node-bfpbd                 250m (6%)     500m (12%)  100Mi (0%)       700Mi (5%)
  kube-system                kube-proxy-7br2g                  50m (1%)      100m (2%)   64Mi (0%)        256Mi (1%)
  kube-system                node-exporter-7kvcm               10m (0%)      20m (0%)    10Mi (0%)        50Mi (0%)
  kube-system                tiller-deploy-56c4cf647b-5vsvb    0 (0%)        0 (0%)      0 (0%)           0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests    Limits
  --------  --------    ------
  cpu       310m (7%)   620m (15%)
  memory    174Mi (1%)  1006Mi (7%)
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 01 Nov 2018 21:31:08 -0400   Thu, 01 Nov 2018 21:31:08 -0400   RouteCreated                 RouteController created a route
  OutOfDisk            False   Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 01 Nov 2018 21:48:50 -0400   Thu, 01 Nov 2018 21:31:06 -0400   KubeletReady                 kubelet is posting ready status

节点 B

Non-terminated Pods:         (4 in total)
  Namespace                  Name                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                  ------------  ----------  ---------------  -------------
  jenkins-test               jenkins-master    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                calico-node-qglbv                     250m (6%)     500m (12%)  100Mi (0%)       700Mi (5%)
  kube-system                kube-proxy-g74ff                      50m (1%)      100m (2%)   64Mi (0%)        256Mi (1%)
  kube-system                node-exporter-bvczb                   10m (0%)      20m (0%)    10Mi (0%)        50Mi (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests    Limits
  --------  --------    ------
  cpu       310m (7%)   620m (15%)
  memory    174Mi (1%)  1006Mi (7%)
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 01 Nov 2018 21:31:06 -0400   Thu, 01 Nov 2018 21:31:06 -0400   RouteCreated                 RouteController created a route
  OutOfDisk            False   Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:30:46 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 01 Nov 2018 21:48:49 -0400   Thu, 01 Nov 2018 21:31:06 -0400   KubeletReady                 kubelet is posting ready status

节点 C

Non-terminated Pods:         (3 in total)
  Namespace                  Name                   CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                   ------------  ----------  ---------------  -------------
  kube-system                calico-node-w9px6      250m (6%)     500m (12%)  100Mi (0%)       700Mi (5%)
  kube-system                kube-proxy-4r2ck       50m (1%)      100m (2%)   64Mi (0%)        256Mi (1%)
  kube-system                node-exporter-r92xs    10m (0%)      20m (0%)    10Mi (0%)        50Mi (0%)
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 01 Nov 2018 21:31:01 -0400   Thu, 01 Nov 2018 21:31:01 -0400   RouteCreated                 RouteController created a route
  OutOfDisk            False   Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:30:49 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:30:49 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:30:49 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:30:49 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 01 Nov 2018 21:48:42 -0400   Thu, 01 Nov 2018 21:31:09 -0400   KubeletReady                 kubelet is posting ready status

节点 D

Non-terminated Pods:         (9 in total)
  Namespace                  Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                                               ------------  ----------  ---------------  -------------
  kube-system                addons-kubernetes-dashboard-8656b6fc5f-68wzm                       50m (1%)      200m (5%)   50Mi (0%)        256Mi (1%)
  kube-system                addons-nginx-ingress-controller-77579b6d64-sqzl7                   100m (2%)     300m (7%)   100Mi (0%)       512Mi (3%)
  kube-system                addons-nginx-ingress-nginx-ingress-k8s-backend-5d6d4598ff-nfzt4    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                calico-node-h2t5b                                                  250m (6%)     500m (12%)  100Mi (0%)       700Mi (5%)
  kube-system                coredns-5c554d9f6f-fnwqq                                           100m (2%)     200m (5%)   15Mi (0%)        80Mi (0%)
  kube-system                kube-proxy-bfhjr                                                   50m (1%)      100m (2%)   64Mi (0%)        256Mi (1%)
  kube-system                metrics-server-7f4cbf557d-985sj                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                node-exporter-frdrd                                                10m (0%)      20m (0%)    10Mi (0%)        50Mi (0%)
  kube-system                vpn-shoot-7bcd5f4bb-88sc7                                          100m (2%)     300m (7%)   128Mi (0%)       512Mi (3%)
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 01 Nov 2018 21:30:54 -0400   Thu, 01 Nov 2018 21:30:54 -0400   RouteCreated                 RouteController created a route
  OutOfDisk            False   Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:30:42 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:30:42 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:30:42 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:30:42 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 01 Nov 2018 21:48:45 -0400   Thu, 01 Nov 2018 21:31:02 -0400   KubeletReady                 kubelet is posting ready status

如所有结果所示,我的节点都很健康并且有充足的资源,每个节点有 4 个 CPU,每个节点有 16GB 内存。

现在,当我尝试在命名空间中部署第二个 statefulSet 时,Pod 仍处于 Pending 状态。描述显示以下消息:

Events:
  Type     Reason             Age                From                Message
  ----     ------             ----               ----                -------
  Warning  FailedScheduling   2m (x123 over 7m)  default-scheduler   pod has unbound PersistentVolumeClaims (repeated 2 times)
  Normal   NotTriggerScaleUp  12s (x26 over 6m)  cluster-autoscaler  pod didn't trigger scale-up (it wouldn't fit if a new node is added)

我还耗尽了 2 个节点,标记了这两个节点并将 nodeSelector 附加到我的 statefulset 以仅部署在这 2 个几乎为空的节点上,但结果是相同的。

我不确定为什么我的 pod 正在尝试 scale-up。这不是意图。任何帮助将不胜感激。

kubectl get po -n jenkins-test -o wide
NAME                                      READY     STATUS    RESTARTS   AGE       IP           NODE                                             NOMINATED NODE
jenkins-agent         0/1       Pending   0          12m       <none>       <none>                                           <none>
jenkins-master        1/1       Running   0          22m       100.96.1.2   shoot--t--csp-worker-hqh6g-z1-6df8f7dc66-bcj6t   <none>

pod has unbound PersistentVolumeClaims ,我觉得是这个问题的关键。关于scale up,Pod是通过'scale from 0 to any instance you defined'创建的。