kubectl exec 命令全部超时
kubectl exec commands are all timing out
我在 GCP 中有 Kubernetes 集群 运行。
直到今天一切正常。 kubectl exec 命令超时：
kubectl exec -it <pod_id> bash
error: Timeout occured
其他一切似乎都正常（例如 kubectl get pods、kubectl logs pod）。
我什至不知道如何调试它。
如有任何帮助，我们将不胜感激。谢谢。
更新详细程度 8
I0110 22:12:15.685414 7793 loader.go:357] Config loaded from file /Users/jahongirr/Documents/Secret Keys/kubeconfig
I0110 22:12:15.686839 7793 round_trippers.go:414] GET https://35.188.50.146/api/v1/namespaces/default/pods/backend-web-deployment-2725661572-spcwv
I0110 22:12:15.686849 7793 round_trippers.go:421] Request Headers:
I0110 22:12:15.686854 7793 round_trippers.go:424] Accept: application/json, */*
I0110 22:12:15.686858 7793 round_trippers.go:424] User-Agent: kubectl/v1.8.4 (darwin/amd64) kubernetes/9befc2b
I0110 22:12:15.686862 7793 round_trippers.go:424] Authorization: Basic YWRtaW46cmlsWHgzbEhXeVdNbDdoNw==
I0110 22:12:16.504372 7793 round_trippers.go:439] Response Status: 200 OK in 817 milliseconds
I0110 22:12:16.504402 7793 round_trippers.go:442] Response Headers:
I0110 22:12:16.504410 7793 round_trippers.go:445] Audit-Id: 146dfe8f-1782-4492-a383-3950c057e6a9
I0110 22:12:16.504416 7793 round_trippers.go:445] Content-Type: application/json
I0110 22:12:16.504422 7793 round_trippers.go:445] Date: Wed, 10 Jan 2018 17:12:16 GMT
I0110 22:12:16.508152 7793 request.go:836] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"backend-web-deployment-2725661572-spcwv","generateName":"backend-web-deployment-2725661572-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/backend-web-deployment-2725661572-spcwv","uid":"5cbfd85a-f5fd-11e7-bd6e-42010a8001b8","resourceVersion":"53394689","creationTimestamp":"2018-01-10T11:56:58Z","labels":{"app":"backend","pod-template-hash":"2725661572"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"backend-web-deployment-2725661572\",\"uid\":\"b5285fc2-f5f4-11e7-aaf7-42010a800150\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"53393600\"}}\n","kubernetes.io/limit-ranger":"LimitRanger plugin set: cpu request for container backend-web-container; cpu request for container cloudsql-proxy"},"ownerReferences":[{"apiVersion":"extensions/v1beta1","kind":"ReplicaSet","name":"backend-web-deployment-2725661572","uid":"b5285fc2-f5f4-11e7-aaf7-42010a800150","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"cloudsql-instance-credentials","secret":{"secretName":"cloudsql-instance-credentials","defaultMode":420}},{"name":"cloudsql","emptyDir":{}},{"name":"ssl-certs","hostPath":{"path":"/etc/ssl/certs","type":""}},{"name":"default-token-wn6gs","secret":{"secretName":"default-token-wn6gs","defaultMode":420}}],"containers":[{"name":"backend-web-container","image":"gcr.io/thematic-flash-786/sd-backend:265","ports":[{"containerPort":8050,"protocol":"TCP"}],"env":[{"name":"POSTGRES_DB_HOST","value":"127.0.0.1"},{"name":"POSTGRES_DB_USER","valueFrom":{"secretKeyRef":{"name":"cloudsql-db-credentials","key":"username"}}},{"name":"POSTGRES_DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"cloudsql-db-credentials","key":"password"}}}],"resources":{"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"default-token-wn6gs","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"httpGet":{"path":"/health","port":8050,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":5,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"},{"name":"cloudsql-proxy","image":"gcr.io/cloudsql-docker/gce-proxy:1.11","command":["/cloud_sql_proxy","--dir=/cloudsql","-instances=thematic-flash-786:us-central1:staging=tcp:5432","-credential_file=/secrets/cloudsql/credentials.json"],"resources":{"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"cloudsql-instance-credentials","readOnly":true,"mountPath":"/secrets/cloudsql"},{"name":"ssl-certs","mountPath":"/etc/ssl/certs"},{"name":"cloudsql","mountPath":"/cloudsql"},{"name":"default-token-wn6gs","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"gke-sd-services-default-pool-6c53cf71-2gtm","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.alpha.kubernetes.io/notReady","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.alpha.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-01-10T11:56:59Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-01-10T12:00:25Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-01-10T11:56:59Z"}],"hostIP":"10.128.0.9","podIP":"10.0.0.4","startTime":"2018-01-10T11:56:59Z","containerStatuses":[{"name":"backend-web-container","state":{"running":{"startedAt":"2018-01-10T11:59:46Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"gcr.io/thematic-flash-786/sd-backend:265","imageID":"docker-pullable://gcr.io/thematic-flash-786/sd-backend@sha256:b11f2a9046969ad24b3c9369e8d7d53b23a592e3fb526840a23792d71ed22985","containerID":"docker://0780bfb5fc9a473f713e881fcd902e7f42d6e432ae30480602be7ef10ec6032e"},{"name":"cloudsql-proxy","state":{"running":{"startedAt":"2018-01-10T11:59:54Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"gcr.io/cloudsql-docker/gce-proxy:1.11","imageID":"docker-pullable://gcr.io/cloudsql-docker/gce-proxy@sha256:5c690349ad8041e7a21eaa63cb078cf13188567e0bfac3b5a914da3483079e2b","containerID":"docker://5d97f3e2355a212772794e2796c2c9570f349bb8772c981805054edb5cfe4519"}],"qosClass":"Burstable"}}
Defaulting container name to backend-web-container.
Use 'kubectl describe pod/backend-web-deployment-2725661572-spcwv' to see all of the containers in this pod.
I0110 22:12:16.542902 7793 round_trippers.go:414] POST https://35.188.50.146/api/v1/namespaces/default/pods/backend-web-deployment-2725661572-spcwv/exec?command=bash&container=backend-web-container&container=backend-web-container&stdin=true&stdout=true&tty=true
I0110 22:12:16.542915 7793 round_trippers.go:421] Request Headers:
I0110 22:12:16.542942 7793 round_trippers.go:424] X-Stream-Protocol-Version: v4.channel.k8s.io
I0110 22:12:16.542947 7793 round_trippers.go:424] X-Stream-Protocol-Version: v3.channel.k8s.io
I0110 22:12:16.542952 7793 round_trippers.go:424] X-Stream-Protocol-Version: v2.channel.k8s.io
I0110 22:12:16.542956 7793 round_trippers.go:424] X-Stream-Protocol-Version: channel.k8s.io
I0110 22:12:16.542966 7793 round_trippers.go:414] POST https://35.188.50.146/api/v1/namespaces/default/pods/backend-web-deployment-2725661572-spcwv/exec?command=bash&container=backend-web-container&container=backend-web-container&stdin=true&stdout=true&tty=true
I0110 22:12:16.542971 7793 round_trippers.go:421] Request Headers:
I0110 22:12:16.542995 7793 round_trippers.go:424] Authorization: Basic YWRtaW46cmlsWHgzbEhOuVdNbDdoNw==
I0110 22:12:16.543000 7793 round_trippers.go:424] X-Stream-Protocol-Version: v4.channel.k8s.io
I0110 22:12:16.543004 7793 round_trippers.go:424] X-Stream-Protocol-Version: v3.channel.k8s.io
I0110 22:12:16.543008 7793 round_trippers.go:424] X-Stream-Protocol-Version: v2.channel.k8s.io
I0110 22:12:16.543012 7793 round_trippers.go:424] X-Stream-Protocol-Version: channel.k8s.io
I0110 22:12:16.543016 7793 round_trippers.go:424] User-Agent: kubectl/v1.8.4 (darwin/amd64) kubernetes/9befc2b
I0110 22:12:17.425534 7793 round_trippers.go:439] Response Status: 101 Switching Protocols in 882 milliseconds
I0110 22:12:17.425556 7793 round_trippers.go:442] Response Headers:
I0110 22:12:17.425561 7793 round_trippers.go:445] Connection: Upgrade
I0110 22:12:17.425565 7793 round_trippers.go:445] Upgrade: SPDY/3.1
I0110 22:12:17.425568 7793 round_trippers.go:445] X-Stream-Protocol-Version: v4.channel.k8s.io
I0110 22:12:17.425572 7793 round_trippers.go:445] Date: Wed, 10 Jan 2018 17:12:17 GMT
I0110 22:12:17.425578 7793 round_trippers.go:439] Response Status: 101 Switching Protocols in 882 milliseconds
I0110 22:12:17.425582 7793 round_trippers.go:442] Response Headers:
I0110 22:12:17.425585 7793 round_trippers.go:445] Connection: Upgrade
I0110 22:12:17.425589 7793 round_trippers.go:445] Upgrade: SPDY/3.1
I0110 22:12:17.425593 7793 round_trippers.go:445] X-Stream-Protocol-Version: v4.channel.k8s.io
I0110 22:12:17.425597 7793 round_trippers.go:445] Date: Wed, 10 Jan 2018 17:12:17 GMT
F0110 22:12:47.838913 7793 helpers.go:120] error: Timeout occured
原来是我的ISP。打开 VPN，它成功了。
我在 GCP 中有 Kubernetes 集群 运行。
直到今天一切正常。 kubectl exec 命令超时：
kubectl exec -it <pod_id> bash
error: Timeout occured
其他一切似乎都正常（例如 kubectl get pods、kubectl logs pod）。
我什至不知道如何调试它。
如有任何帮助，我们将不胜感激。谢谢。
更新详细程度 8
I0110 22:12:15.685414 7793 loader.go:357] Config loaded from file /Users/jahongirr/Documents/Secret Keys/kubeconfig
I0110 22:12:15.686839 7793 round_trippers.go:414] GET https://35.188.50.146/api/v1/namespaces/default/pods/backend-web-deployment-2725661572-spcwv
I0110 22:12:15.686849 7793 round_trippers.go:421] Request Headers:
I0110 22:12:15.686854 7793 round_trippers.go:424] Accept: application/json, */*
I0110 22:12:15.686858 7793 round_trippers.go:424] User-Agent: kubectl/v1.8.4 (darwin/amd64) kubernetes/9befc2b
I0110 22:12:15.686862 7793 round_trippers.go:424] Authorization: Basic YWRtaW46cmlsWHgzbEhXeVdNbDdoNw==
I0110 22:12:16.504372 7793 round_trippers.go:439] Response Status: 200 OK in 817 milliseconds
I0110 22:12:16.504402 7793 round_trippers.go:442] Response Headers:
I0110 22:12:16.504410 7793 round_trippers.go:445] Audit-Id: 146dfe8f-1782-4492-a383-3950c057e6a9
I0110 22:12:16.504416 7793 round_trippers.go:445] Content-Type: application/json
I0110 22:12:16.504422 7793 round_trippers.go:445] Date: Wed, 10 Jan 2018 17:12:16 GMT
I0110 22:12:16.508152 7793 request.go:836] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"backend-web-deployment-2725661572-spcwv","generateName":"backend-web-deployment-2725661572-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/backend-web-deployment-2725661572-spcwv","uid":"5cbfd85a-f5fd-11e7-bd6e-42010a8001b8","resourceVersion":"53394689","creationTimestamp":"2018-01-10T11:56:58Z","labels":{"app":"backend","pod-template-hash":"2725661572"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"backend-web-deployment-2725661572\",\"uid\":\"b5285fc2-f5f4-11e7-aaf7-42010a800150\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"53393600\"}}\n","kubernetes.io/limit-ranger":"LimitRanger plugin set: cpu request for container backend-web-container; cpu request for container cloudsql-proxy"},"ownerReferences":[{"apiVersion":"extensions/v1beta1","kind":"ReplicaSet","name":"backend-web-deployment-2725661572","uid":"b5285fc2-f5f4-11e7-aaf7-42010a800150","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"cloudsql-instance-credentials","secret":{"secretName":"cloudsql-instance-credentials","defaultMode":420}},{"name":"cloudsql","emptyDir":{}},{"name":"ssl-certs","hostPath":{"path":"/etc/ssl/certs","type":""}},{"name":"default-token-wn6gs","secret":{"secretName":"default-token-wn6gs","defaultMode":420}}],"containers":[{"name":"backend-web-container","image":"gcr.io/thematic-flash-786/sd-backend:265","ports":[{"containerPort":8050,"protocol":"TCP"}],"env":[{"name":"POSTGRES_DB_HOST","value":"127.0.0.1"},{"name":"POSTGRES_DB_USER","valueFrom":{"secretKeyRef":{"name":"cloudsql-db-credentials","key":"username"}}},{"name":"POSTGRES_DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"cloudsql-db-credentials","key":"password"}}}],"resources":{"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"default-token-wn6gs","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"httpGet":{"path":"/health","port":8050,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":5,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"},{"name":"cloudsql-proxy","image":"gcr.io/cloudsql-docker/gce-proxy:1.11","command":["/cloud_sql_proxy","--dir=/cloudsql","-instances=thematic-flash-786:us-central1:staging=tcp:5432","-credential_file=/secrets/cloudsql/credentials.json"],"resources":{"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"cloudsql-instance-credentials","readOnly":true,"mountPath":"/secrets/cloudsql"},{"name":"ssl-certs","mountPath":"/etc/ssl/certs"},{"name":"cloudsql","mountPath":"/cloudsql"},{"name":"default-token-wn6gs","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"gke-sd-services-default-pool-6c53cf71-2gtm","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.alpha.kubernetes.io/notReady","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.alpha.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-01-10T11:56:59Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-01-10T12:00:25Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-01-10T11:56:59Z"}],"hostIP":"10.128.0.9","podIP":"10.0.0.4","startTime":"2018-01-10T11:56:59Z","containerStatuses":[{"name":"backend-web-container","state":{"running":{"startedAt":"2018-01-10T11:59:46Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"gcr.io/thematic-flash-786/sd-backend:265","imageID":"docker-pullable://gcr.io/thematic-flash-786/sd-backend@sha256:b11f2a9046969ad24b3c9369e8d7d53b23a592e3fb526840a23792d71ed22985","containerID":"docker://0780bfb5fc9a473f713e881fcd902e7f42d6e432ae30480602be7ef10ec6032e"},{"name":"cloudsql-proxy","state":{"running":{"startedAt":"2018-01-10T11:59:54Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"gcr.io/cloudsql-docker/gce-proxy:1.11","imageID":"docker-pullable://gcr.io/cloudsql-docker/gce-proxy@sha256:5c690349ad8041e7a21eaa63cb078cf13188567e0bfac3b5a914da3483079e2b","containerID":"docker://5d97f3e2355a212772794e2796c2c9570f349bb8772c981805054edb5cfe4519"}],"qosClass":"Burstable"}}
Defaulting container name to backend-web-container.
Use 'kubectl describe pod/backend-web-deployment-2725661572-spcwv' to see all of the containers in this pod.
I0110 22:12:16.542902 7793 round_trippers.go:414] POST https://35.188.50.146/api/v1/namespaces/default/pods/backend-web-deployment-2725661572-spcwv/exec?command=bash&container=backend-web-container&container=backend-web-container&stdin=true&stdout=true&tty=true
I0110 22:12:16.542915 7793 round_trippers.go:421] Request Headers:
I0110 22:12:16.542942 7793 round_trippers.go:424] X-Stream-Protocol-Version: v4.channel.k8s.io
I0110 22:12:16.542947 7793 round_trippers.go:424] X-Stream-Protocol-Version: v3.channel.k8s.io
I0110 22:12:16.542952 7793 round_trippers.go:424] X-Stream-Protocol-Version: v2.channel.k8s.io
I0110 22:12:16.542956 7793 round_trippers.go:424] X-Stream-Protocol-Version: channel.k8s.io
I0110 22:12:16.542966 7793 round_trippers.go:414] POST https://35.188.50.146/api/v1/namespaces/default/pods/backend-web-deployment-2725661572-spcwv/exec?command=bash&container=backend-web-container&container=backend-web-container&stdin=true&stdout=true&tty=true
I0110 22:12:16.542971 7793 round_trippers.go:421] Request Headers:
I0110 22:12:16.542995 7793 round_trippers.go:424] Authorization: Basic YWRtaW46cmlsWHgzbEhOuVdNbDdoNw==
I0110 22:12:16.543000 7793 round_trippers.go:424] X-Stream-Protocol-Version: v4.channel.k8s.io
I0110 22:12:16.543004 7793 round_trippers.go:424] X-Stream-Protocol-Version: v3.channel.k8s.io
I0110 22:12:16.543008 7793 round_trippers.go:424] X-Stream-Protocol-Version: v2.channel.k8s.io
I0110 22:12:16.543012 7793 round_trippers.go:424] X-Stream-Protocol-Version: channel.k8s.io
I0110 22:12:16.543016 7793 round_trippers.go:424] User-Agent: kubectl/v1.8.4 (darwin/amd64) kubernetes/9befc2b
I0110 22:12:17.425534 7793 round_trippers.go:439] Response Status: 101 Switching Protocols in 882 milliseconds
I0110 22:12:17.425556 7793 round_trippers.go:442] Response Headers:
I0110 22:12:17.425561 7793 round_trippers.go:445] Connection: Upgrade
I0110 22:12:17.425565 7793 round_trippers.go:445] Upgrade: SPDY/3.1
I0110 22:12:17.425568 7793 round_trippers.go:445] X-Stream-Protocol-Version: v4.channel.k8s.io
I0110 22:12:17.425572 7793 round_trippers.go:445] Date: Wed, 10 Jan 2018 17:12:17 GMT
I0110 22:12:17.425578 7793 round_trippers.go:439] Response Status: 101 Switching Protocols in 882 milliseconds
I0110 22:12:17.425582 7793 round_trippers.go:442] Response Headers:
I0110 22:12:17.425585 7793 round_trippers.go:445] Connection: Upgrade
I0110 22:12:17.425589 7793 round_trippers.go:445] Upgrade: SPDY/3.1
I0110 22:12:17.425593 7793 round_trippers.go:445] X-Stream-Protocol-Version: v4.channel.k8s.io
I0110 22:12:17.425597 7793 round_trippers.go:445] Date: Wed, 10 Jan 2018 17:12:17 GMT
F0110 22:12:47.838913 7793 helpers.go:120] error: Timeout occured
原来是我的ISP。打开 VPN，它成功了。