Kubernetes“Windowspods”无法相互通信 | “无法解析主机” | "failed to connect Network unreachable"

Kubernetes " Windows pods " not able to communicate with each other | “could not resolve host” | "failed to connect Network unreachable"

我部署了三个后端服务到 kubernetes windows pods 以确保它们相互通信。但是,我发现尽管容器、服务、dns 和端点都可用并且 运行 但是当我尝试从一个容器访问任何服务(内部或外部)到另一个时,它仍然无法解析 dns并收到“无法解析主机”或 "failed to connect Network unreachable"

尝试通过(资源服务)访问人员配备或聚合器服务时,>收到以下错误:

C:\WINDOWS\system32>kubectl exec -it resources-deployment-6b98d47b5b-ts674 cmd

Microsoft Windows [Version 10.0.17763.864] (c) 2018 Microsoft Corporation. All rights reserved.

C:\app\resources>curl http://172.20.101.110:8081/swagger/index.html curl: (7) Failed to connect to 172.20.101.110 port 8081: Network unreachable

C:\app\resources>curl http://aggregator-service:8081/swagger/index.html curl: (6) Could not resolve host: aggregator-service

C:\app\resources>curl http://staffing-service:8080/swagger/index.html curl: (6) Could not resolve host: staffing-service

C:\app\resources>curl http://172.20.5.156:8080/swagger/index.html curl: (7) Failed to connect to 172.20.5.156 port 8080: Network unreachable

C:\app\resources>ping 10.91.24.250

Pinging 10.91.24.250 with 32 bytes of data: PING: transmit failed. General failure.

代码很简单,这里是完整的细节。

它是一个混合环境,有两个 windows 节点和两个 linux 节点。但是目前的要求是仅在 windows 个节点上部署以下应用程序。基础设施部署在AWS提供的EKS(Elastic Kubernetes Service)上。

三个后端(aspnet)服务是:-

  1. Staffing.yaml 类型为 LoadBalancer 的文件。

  2. Resources.Yaml 类型为 NodePort

  3. Aggregator.Yaml 类型为 NodePort

要求是 Resources API 应该能够连接到 Aggregator 和 Staffing 服务。 Staffing API 也可以连接到任何服务(资源和聚合器)

Deployment and Services YAML file are:- Staffing.yaml file

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: staffing-deployment
      #namespace: staffing-kube-system
      labels:
        app: staffing-app
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: staffing-app
      template:
        metadata:
          labels:
            app: staffing-app
        spec:
          nodeSelector:
           beta.kubernetes.io/os: windows
           #failure-domain.beta.kubernetes.io/zone: us-east-1a
          containers:
          - name: staffing-app
            ports:
             - name: http
               containerPort: 8080
            image: test/staffing:v1
    ##########################
    #Staffing.service.yaml file
    --- 
    apiVersion: v1
    kind: Service
    metadata: 
      labels: 
        app: staffing-app
      name: staffing-service
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    spec: 
      ports: 
        - 
          name: http
          #nodePort: 30475
          port: 8080
          protocol: TCP
          targetPort: 8080
        - 
          name: https
          #nodePort: 30476
          port: 443
          protocol: TCP
          targetPort: 44346
      selector: 
        app: staffing-app
      type: LoadBalancer

resources.service.deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: resources-deployment
  #namespace: staffing-kube-system
  labels:
    app: resources-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: resources-app
  template:
    metadata:
      labels:
        app: resources-app
    spec:
      nodeSelector:
       beta.kubernetes.io/os: windows
       #failure-domain.beta.kubernetes.io/zone: us-east-1a
      containers:
      - name: resources-app
        ports:
         - name: http
           containerPort: 8001
        image: test/resources:v1
################
#resources.service.yaml
apiVersion: v1
kind: Service
metadata: 
  labels: 
    app: resources-app
  name: resources-service
 spec: 
  ports: 
    - 
      name: http
      #nodePort: 30475
      port: 8001
      protocol: TCP
      targetPort: 8001
    - 
      name: https
      #nodePort: 30476
      port: 443
      protocol: TCP
      targetPort: 44345
  selector: 
    app: resources-app
  type: NodePort

aggregator.service.deplpoyment.yaml

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: aggegator-deployment
      #namespace: staffing-kube-system
      labels:
        app: aggegator-app
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: aggregator-app
      template:
        metadata:
          labels:
            app: aggregator-app
        spec:
          nodeSelector:
           beta.kubernetes.io/os: windows
           #failure-domain.beta.kubernetes.io/zone: us-east-1a
          containers:
          - name: aggregator-app
            ports:
             - name: http
               containerPort: 8001
            image: test/aggregator:v1
 ################
    #aggregator.service.yaml
        apiVersion: v1
        kind: Service
        metadata: 
          labels: 
            app: resources-app
          name: resources-service
         spec: 
          ports: 
            - 
              name: http
              #nodePort: 30475
              port: 8001
              protocol: TCP
              targetPort: 8001
            - 
              name: https
              #nodePort: 30476
              port: 443
              protocol: TCP
              targetPort: 44345
          selector: 
            app: resources-app
          type: NodePort

Output of Pods and services

C:\WINDOWS\system32>kubectl get pods -o wide
NAME                                    READY   STATUS    RESTARTS   AGE     IP             NODE                           NOMINATED NODE   READINESS GATES
aggegator-deployment-64497699d-792dd    1/1     Running   0          2d12h   10.91.25.210   ip-10-91-25-125.ec2.internal   <none>           <none>
aggegator-deployment-64497699d-tfsww    1/1     Running   0          2d12h   10.91.24.250   ip-10-91-24-148.ec2.internal   <none>           <none>
resources-deployment-6b98d47b5b-7x8fv   1/1     Running   0          2d13h   10.91.25.197   ip-10-91-25-125.ec2.internal   <none>           <none>
resources-deployment-6b98d47b5b-ts674   1/1     Running   0          2d13h   10.91.24.122   ip-10-91-24-148.ec2.internal   <none>           <none>
staffing-deployment-797464966d-4j4nc    1/1     Running   0          2d16h   10.91.24.224   ip-10-91-24-148.ec2.internal   <none>           <none>
staffing-deployment-797464966d-mrw7x    1/1     Running   0          2d17h   10.91.24.113   ip-10-91-24-148.ec2.internal   <none>           <none>




C:\WINDOWS\system32>kubectl get services -o wide
NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                        PORT(S)                        AGE     SELECTOR
aggregator-service           NodePort       172.20.101.110   <none>                                                                             8081:32246/TCP,443:30240/TCP   2d12h   app=aggregator-app
kubernetes                   ClusterIP      172.20.0.1       <none>                                                                             443/TCP                        21d     <none>
resources-service            NodePort       172.20.228.37    <none>                                                                             8001:31365/TCP,443:31686/TCP   2d13h   app=resources-app
staffing-service             LoadBalancer   172.20.5.156     internal-af561cd570d0d11eab0c00a891642f9b-874414568.us-east-1.elb.amazonaws.com    8080:30010/TCP,443:31327/TCP   2d17h   app=staffing-app



C:\WINDOWS\system32>kubectl get pods --selector="app=resources-app"
NAME                                    READY   STATUS    RESTARTS   AGE
resources-deployment-6b98d47b5b-7x8fv   1/1     Running   0          2d21h
resources-deployment-6b98d47b5b-ts674   1/1     Running   0          2d21h

C:\WINDOWS\system32>kubectl get pods --selector="app=aggregator-app"
NAME                                   READY   STATUS    RESTARTS   AGE
aggegator-deployment-64497699d-792dd   1/1     Running   0          2d20h
aggegator-deployment-64497699d-tfsww   1/1     Running   0          2d20h

C:\WINDOWS\system32>kubectl get pods --selector="app=staffing-app"
NAME                                   READY   STATUS    RESTARTS   AGE
staffing-deployment-797464966d-4j4nc   1/1     Running   0          3d
staffing-deployment-797464966d-mrw7x   1/1     Running   0          3d1h


C:\WINDOWS\system32>kubectl get endpoints
NAME                         ENDPOINTS                                                             AGE
aggregator-service           10.91.24.250:8081,10.91.25.210:8081,10.91.24.250:44347 + 1 more...    2d20h
kubernetes                   10.91.24.167:443,10.91.25.120:443                                     22d
resources-service            10.91.24.122:8001,10.91.25.197:8001,10.91.24.122:44345 + 1 more...    2d21h
staffing-service             10.91.24.113:44346,10.91.24.224:44346,10.91.24.113:8080 + 1 more...   3d2h

> Kubectl logs

C:\WINDOWS\system32>kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-8455f84f99-d6v75   1/1     Running   0          21d
coredns-8455f84f99-jf7tz   1/1     Running   0          21d

C:\WINDOWS\system32>kubectl logs coredns-8455f84f99-jf7tz  -n kube-system
.:53
2019-11-03T14:27:30.347Z [INFO] CoreDNS-1.3.1
2019-11-03T14:27:30.347Z [INFO] linux/amd64, go1.10.8, 6b56a9c9
CoreDNS-1.3.1
linux/amd64, go1.10.8, 6b56a9c9
2019-11-03T14:27:30.347Z [INFO] plugin/reload: Running configuration MD5 = 86a350d0c3832da65c10531b445c45d5
[INFO] Reloading
2019-11-25T07:00:09.649Z [INFO] plugin/reload: Running configuration MD5 = ebe4e401326d5dac294cde91064e2e76
[INFO] Reloading complete
2019-11-25T11:38:44.107Z [INFO] 10.91.24.148:57354 - 8232 "A IN . udp 17 false 512" NOERROR qr,rd,ra 92 0.000084547s
2019-11-25T11:38:44.113Z [INFO] 10.91.24.148:57364 - 33842 "A IN . udp 17 false 512" NOERROR qr,rd,ra 92 0.000059248s
2019-11-25T11:39:15.287Z [INFO] 10.91.24.148:57366 - 23694 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.003154278s
2019-11-25T11:39:15.331Z [INFO] 10.91.24.148:57366 - 23694 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.069842673s
2019-11-25T11:39:15.615Z [INFO] 10.91.25.125:61221 - 24722 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.000102425s
2019-11-25T11:39:17.575Z [INFO] 10.91.25.125:61223 - 8379 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.000096859s
2019-11-25T11:39:18.925Z [INFO] 10.91.25.125:61239 - 49217 "A IN settings-win.data.microsoft.com. udp 49 false 512" NOERROR qr,rd,ra 176 0.030653295s
2019-11-25T11:39:19.006Z [INFO] 10.91.25.125:61239 - 49217 "A IN settings-win.data.microsoft.com. udp 49 false 512" NOERROR qr,rd,ra 176 0.142995164s
2019-11-25T11:39:19.205Z [INFO] 10.91.25.125:61241 - 25186 "A IN settings-win.data.microsoft.com. udp 49 false 512" NOERROR qr,rd,ra 176 0.000088335s
2019-11-25T11:39:43.422Z [INFO] 10.91.25.125:61257 - 26043 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.013510742s
2019-11-25T11:39:44.129Z [INFO] 10.91.24.148:57368 - 382 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.013958603s
2019-11-25T11:39:44.270Z [INFO] 10.91.25.125:61261 - 53992 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.014368787s
2019-11-25T11:39:44.981Z [INFO] 10.91.24.148:57370 - 41136 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.014143535s
2019-11-25T11:40:15.649Z [INFO] 10.91.25.125:61231 - 53428 "A IN ctldl.windowsupdate.com. udp 41 false 512" NOERROR qr,rd,ra 355 0.013625947s
2019-11-25T11:40:45.838Z [INFO] 10.91.24.148:57372 - 36065 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.013761424s
2019-11-25T11:40:46.688Z [INFO] 10.91.24.148:57382 - 26755 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.01357897s
2019-11-25T11:40:51.896Z [INFO] 10.91.24.148:57388 - 607 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.033610722s
2019-11-25T11:40:52.006Z [INFO] 10.91.24.148:57388 - 607 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.11348618s
2019-11-25T11:41:46.839Z [INFO] 10.91.25.125:61227 - 9202 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.013585559s
2019-11-25T11:41:47.553Z [INFO] 10.91.24.148:57339 - 12070 "A IN . udp 17 false 512" NOERROR qr,rd,ra 92 0.001922224s
2019-11-25T11:41:48.403Z [INFO] 10.91.24.148:57341 - 52311 "A IN . udp 17 false 512" NOERROR qr,rd,ra 92 0.000091287s
2019-11-25T11:42:04.827Z [INFO] 10.91.24.148:57343 - 23836 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.028061963s
2019-11-25T11:42:04.827Z [INFO] 10.91.24.148:57343 - 23836 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.004729734s
2019-11-25T11:42:13.855Z [hINFO] 10.91.24.148:57344 - 62298 "A IN ctldl.windowsupdate.com. udp 41 false 512" NOERROR qr,rd,ra 540 0.012435227s

如有任何帮助,我们将不胜感激。谢谢

问题出在服务定义中,其中 Kubernetes 文件中定义的端口与代码中定义的端口冲突。