Kubernetes - 通过服务名调用微服务
Kubernetes - Calling Microservice by Service Name
我在 K8S 集群上部署了两个微服务（本地在 3 个虚拟机上 - 1 个主节点和 2 个工作节点）：
1-货币兑换微服务
2-货币转换微服务
我正在尝试使用服务名称从 currency-conversion 调用 currency-exchange 微服务：
http:///currency-exchange:8000.
它returns错误如下：
{"timestamp":"2021-02-17T08:38:25.590+0000","status":500,"error":"Internal Server Error","message":"currency-exchange executing GET http://currency-exchange:8000/currency-exchange/from/EUR/to/INR","path":"/currency-conversion/from/EUR/to/INR/quantity/10"}
我正在使用 Kubernetes，CentOS8 使用带有设置 FELIX_IPTABLESBACKEND=NFT 的 Calico CNI，基于 this link 以促进 POD-TO-POD 通信。
当前可用的服务：
[root@k8s-master ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
currency-conversion NodePort 10.106.70.108 <none> 8100:32470/TCP 3h40m
currency-exchange NodePort 10.110.232.189 <none> 8000:31776/TCP 3h41m
Pods:
[root@k8s-master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
currency-conversion-86d9bc4698-rxdkh 1/1 Running 0 5h45m 192.168.212.125 worker-node-1 <none> <none>
currency-exchange-c79ff888b-c8sdd 1/1 Running 0 5h44m 192.168.19.160 worker-node-2 <none> <none>
currency-exchange-c79ff888b-nfqpx 1/1 Running 0 5h44m 192.168.212.65 worker-node-1 <none> <none>
可用的 CoreDNS Pods 列表：
[root@k8s-master ~]# kubectl get pods -o wide -n kube-system | grep coredns
coredns-74ff55c5b-9x5qm 1/1 Running 8 25d 192.168.235.218 k8s-master <none> <none>
coredns-74ff55c5b-zkkn7 1/1 Running 8 25d 192.168.235.220 k8s-master <none> <none>
列出所有 ENV 变量：
[root@k8s-master ~]# kubectl exec -it currency-conversion-86d9bc4698-rxdkh -- printenv
HOSTNAME=currency-conversion-86d9bc4698-rxdkh
CURRENCY_EXCHANGE_SERVICE_HOST=http://currency-exchange
KUBERNETES_SERVICE_HOST=10.96.0.1
CURRENCY_EXCHANGE_SERVICE_PORT=8000
........
nslookup kubernetes.default 执行命令：
[root@k8s-master ~]# kubectl exec -it currency-conversion-86d9bc4698-rxdkh -- nslookup kubernetes.default
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'kubernetes.default': Try again
command terminated with exit code 1
人们如何解决这样的问题？他们 configure/tweak DNS 是否可以作为服务注册中心正常工作？
提前致谢
已编辑：
[root@k8s-master ~]# kubectl describe service currency-conversion
Name: currency-conversion
Namespace: default
Labels: app=currency-conversion
Annotations: <none>
Selector: app=currency-conversion
Type: NodePort
IP Families: <none>
IP: 10.106.70.108
IPs: 10.106.70.108
Port: <unset> 8100/TCP
TargetPort: 8100/TCP
NodePort: <unset> 32470/TCP
Endpoints: 192.168.212.125:8100
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
[root@k8s-master ~]# kubectl describe service currency-exchange
Name: currency-exchange
Namespace: default
Labels: app=currency-exchange
Annotations: <none>
Selector: app=currency-exchange
Type: NodePort
IP Families: <none>
IP: 10.110.232.189
IPs: 10.110.232.189
Port: <unset> 8000/TCP
TargetPort: 8000/TCP
NodePort: <unset> 31776/TCP
Endpoints: 192.168.19.160:8000,192.168.212.65:8000
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
我刚刚发现所有 coredns PODS 中的错误行为，很多超时：
[root@k8s-master ~]# kubectl logs coredns-74ff55c5b-zkkn7 -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:59744->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:53400->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:58465->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:58197->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:57794->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:43345->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:57361->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:51716->192.168.100.1:53: i/o timeout
我如何开始跟踪问题？
额外详情：
[root@k8s-master ~]# kubectl exec -i -t currency-conversion-86d9bc4698-rxdkh -- sh
/ # wget http://currency-exchange:8000/currency-exchange/from/EUR/to/INR
wget: bad address 'currency-exchange:8000'
在我看来，您错误地设置了 CNI 覆盖网络。我检查了你之前的问题以验证节点的 ip 地址，在我看来你的 pod 网络与你的主机网络重叠：
Kubernetes pod-network-cidr 是 Kubernetes 集群中所有 pods 的 IP prefix。此范围不得与您 VPC
中的其他网络冲突
Kubernetes pod 网络 documentation 对此也有描述：
Take care that your Pod network must not overlap with any of the host
networks: you are likely to see problems if there is any overlap. (If
you find a collision between your network plugin's preferred Pod
network and some of your host networks, you should think of a suitable
CIDR block to use instead, then use that during kubeadm init with
--pod-network-cidr and as a replacement in your network plugin's
YAML).
创建cluster时在calico指令中也提到了这一点：
Note: If 192.168.0.0/16 is already in use within your network you must select a different pod network CIDR, replacing 192.168.0.0/16 in
the above command.
PS。您始终可以 wget 从 here.
卷曲
我在 K8S 集群上部署了两个微服务（本地在 3 个虚拟机上 - 1 个主节点和 2 个工作节点）：
1-货币兑换微服务
2-货币转换微服务
我正在尝试使用服务名称从 currency-conversion 调用 currency-exchange 微服务：
http:///currency-exchange:8000.
它returns错误如下：
{"timestamp":"2021-02-17T08:38:25.590+0000","status":500,"error":"Internal Server Error","message":"currency-exchange executing GET http://currency-exchange:8000/currency-exchange/from/EUR/to/INR","path":"/currency-conversion/from/EUR/to/INR/quantity/10"}
我正在使用 Kubernetes，CentOS8 使用带有设置 FELIX_IPTABLESBACKEND=NFT 的 Calico CNI，基于 this link 以促进 POD-TO-POD 通信。
当前可用的服务：
[root@k8s-master ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
currency-conversion NodePort 10.106.70.108 <none> 8100:32470/TCP 3h40m
currency-exchange NodePort 10.110.232.189 <none> 8000:31776/TCP 3h41m
Pods:
[root@k8s-master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
currency-conversion-86d9bc4698-rxdkh 1/1 Running 0 5h45m 192.168.212.125 worker-node-1 <none> <none>
currency-exchange-c79ff888b-c8sdd 1/1 Running 0 5h44m 192.168.19.160 worker-node-2 <none> <none>
currency-exchange-c79ff888b-nfqpx 1/1 Running 0 5h44m 192.168.212.65 worker-node-1 <none> <none>
可用的 CoreDNS Pods 列表：
[root@k8s-master ~]# kubectl get pods -o wide -n kube-system | grep coredns
coredns-74ff55c5b-9x5qm 1/1 Running 8 25d 192.168.235.218 k8s-master <none> <none>
coredns-74ff55c5b-zkkn7 1/1 Running 8 25d 192.168.235.220 k8s-master <none> <none>
列出所有 ENV 变量：
[root@k8s-master ~]# kubectl exec -it currency-conversion-86d9bc4698-rxdkh -- printenv
HOSTNAME=currency-conversion-86d9bc4698-rxdkh
CURRENCY_EXCHANGE_SERVICE_HOST=http://currency-exchange
KUBERNETES_SERVICE_HOST=10.96.0.1
CURRENCY_EXCHANGE_SERVICE_PORT=8000
........
nslookup kubernetes.default 执行命令：
[root@k8s-master ~]# kubectl exec -it currency-conversion-86d9bc4698-rxdkh -- nslookup kubernetes.default
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'kubernetes.default': Try again
command terminated with exit code 1
人们如何解决这样的问题？他们 configure/tweak DNS 是否可以作为服务注册中心正常工作？
提前致谢
已编辑：
[root@k8s-master ~]# kubectl describe service currency-conversion
Name: currency-conversion
Namespace: default
Labels: app=currency-conversion
Annotations: <none>
Selector: app=currency-conversion
Type: NodePort
IP Families: <none>
IP: 10.106.70.108
IPs: 10.106.70.108
Port: <unset> 8100/TCP
TargetPort: 8100/TCP
NodePort: <unset> 32470/TCP
Endpoints: 192.168.212.125:8100
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
[root@k8s-master ~]# kubectl describe service currency-exchange
Name: currency-exchange
Namespace: default
Labels: app=currency-exchange
Annotations: <none>
Selector: app=currency-exchange
Type: NodePort
IP Families: <none>
IP: 10.110.232.189
IPs: 10.110.232.189
Port: <unset> 8000/TCP
TargetPort: 8000/TCP
NodePort: <unset> 31776/TCP
Endpoints: 192.168.19.160:8000,192.168.212.65:8000
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
我刚刚发现所有 coredns PODS 中的错误行为，很多超时：
[root@k8s-master ~]# kubectl logs coredns-74ff55c5b-zkkn7 -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:59744->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:53400->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:58465->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:58197->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:57794->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:43345->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:57361->192.168.100.1:53: i/o timeout
[ERROR] plugin/errors: 2 6675909625369157619.3582573715596351475. HINFO: read udp 192.168.235.221:51716->192.168.100.1:53: i/o timeout
我如何开始跟踪问题？
额外详情：
[root@k8s-master ~]# kubectl exec -i -t currency-conversion-86d9bc4698-rxdkh -- sh
/ # wget http://currency-exchange:8000/currency-exchange/from/EUR/to/INR
wget: bad address 'currency-exchange:8000'
在我看来，您错误地设置了 CNI 覆盖网络。我检查了你之前的问题以验证节点的 ip 地址，在我看来你的 pod 网络与你的主机网络重叠：
Kubernetes pod-network-cidr 是 Kubernetes 集群中所有 pods 的 IP prefix。此范围不得与您 VPC
Kubernetes pod 网络 documentation 对此也有描述：
Take care that your Pod network must not overlap with any of the host networks: you are likely to see problems if there is any overlap. (If you find a collision between your network plugin's preferred Pod network and some of your host networks, you should think of a suitable CIDR block to use instead, then use that during
kubeadm initwith--pod-network-cidrand as a replacement in your network plugin's YAML).
创建cluster时在calico指令中也提到了这一点：
Note: If 192.168.0.0/16 is already in use within your network you must select a different pod network CIDR, replacing 192.168.0.0/16 in the above command.
PS。您始终可以 wget 从 here.