Kubernetes 服务通信 isse - Kubedns
Kubernetes service communication isse - Kubedns
我有两个 pods 映射到两个服务，并且 运行 在我的笔记本电脑中使用虚拟机虚拟机。我有 kube dns 工作。一个 pod 是网络服务，另一个是 mongodb。
webapp pod 规格如下
spec:
containers:
- resources:
limits:
cpu: 0.5
.
.
name: wsemp
ports:
- containerPort: 8080
# name: wsemp
#command: ["java","-Dspring.data.mongodb.uri=mongodb://192.168.6.103:30061/microservices", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
command: ["java","-Dspring.data.mongodb.uri=mongodb://mongoservice/microservices", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
对应服务的规格
apiVersion: v1
kind: Service
metadata:
labels:
name: webappservice
name: webappservice
spec:
ports:
- port: 8080
nodePort: 30062
targetPort: 8080
protocol: TCP
type: NodePort
selector:
name: webapp
Mongodb 吊舱规格
apiVersion: v1
kind: Pod
metadata:
name: mongodb
labels:
name: mongodb
spec:
containers:
.
.
name: mongodb
ports:
- containerPort: 27017
Mongodb 服务规范
apiVersion: v1
kind: Service
metadata:
labels:
name: mongodb
name: mongoservice
spec:
ports:
- port: 27017
nodePort: 30061
targetPort: 27017
protocol: TCP
type: NodePort
selector:
name: mongodb
评论后更新的目标端口在服务中
问题
webapp 启动时无法连接到 mongoservice 端口并在启动时出现此错误
Exception in monitor thread while connecting to server mongoservice:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[mongodb-driver-core-3.2.2.jar!/:na]
at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:114) ~[mongodb-driver-core-3.2.2.jar!/:na]
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:128) ~[mongodb-driver-core-3.2.2.jar!/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_111]
描述 svc
kubectl describe svc mongoservice
Name: mongoservice
Namespace: default
Labels: name=mongodb
Selector: name=mongodb
Type: NodePort
IP: 10.254.146.189
Port: <unset> 27017/TCP
NodePort: <unset> 30061/TCP
Endpoints: 172.17.99.2:27017
Session Affinity: None
No events.
kubectl describe svc webappservice
Name: webappservice
Namespace: default
Labels: name=webappservice
Selector: name=webapp
Type: NodePort
IP: 10.254.112.121
Port: <unset> 8080/TCP
NodePort: <unset> 30062/TCP
Endpoints: 172.17.99.3:8080
Session Affinity: None
No events.
调试
root@webapp:/# nslookup mongoservice
Server: 10.254.0.2
Address: 10.254.0.2#53
Non-authoritative answer:
Name: mongoservice.default.svc.cluster.local
Address: 10.254.146.189
root@webapp:/# curl 10.254.146.189:27017
curl: (7) Failed to connect to 10.254.146.189 port 27017: Connection refused
root@webapp:/# curl mongoservice:27017
curl: (7) Failed to connect to mongoservice port 27017: Connection refused
sudo iptables-save | grep webapp
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/webappservice:" -m tcp --dport 30062 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/webappservice:" -m tcp --dport 30062 -j KUBE-SVC-NQBDRRKQULANV7O3
-A KUBE-SEP-IE7EBTQCN7T6HXC4 -s 172.17.99.3/32 -m comment --comment "default/webappservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-IE7EBTQCN7T6HXC4 -p tcp -m comment --comment "default/webappservice:" -m tcp -j DNAT --to-destination 172.17.99.3:8080
-A KUBE-SERVICES -d 10.254.217.24/32 -p tcp -m comment --comment "default/webappservice: cluster IP" -m tcp --dport 8080 -j KUBE-SVC-NQBDRRKQULANV7O3
-A KUBE-SVC-NQBDRRKQULANV7O3 -m comment --comment "default/webappservice:" -j KUBE-SEP-IE7EBTQCN7T6HXC4
$ curl 10.254.217.24:8080
{"timestamp":1486678423757,"status":404,"error":"Not Found","message":"No message available","path":"/"}[osboxes@kube-node1 ~]$
sudo iptables-save | grep mongodb
[osboxes@osboxes ~]$ sudo iptables-save | grep mongo
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/mongoservice:" -m tcp --dport 30061 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/mongoservice:" -m tcp --dport 30061 -j KUBE-SVC-2HQWGC3WSIBZF7CN
-A KUBE-SEP-FVWOWAWXXVAVIQ5O -s 172.17.99.2/32 -m comment --comment "default/mongoservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-FVWOWAWXXVAVIQ5O -p tcp -m comment --comment "default/mongoservice:" -m tcp -j DNAT --to-destination 172.17.99.2:27017
-A KUBE-SERVICES -d 10.254.146.189/32 -p tcp -m comment --comment "default/mongoservice: cluster IP" -m tcp --dport 27017 -j KUBE-SVC-2HQWGC3WSIBZF7CN
-A KUBE-SVC-2HQWGC3WSIBZF7CN -m comment --comment "default/mongoservice:" -j KUBE-SEP-FVWOWAWXXVAVIQ5O
[osboxes@osboxes ~]$ sudo curl 10.254.146.189:8080
^C[osboxes@osboxes ~]$ sudo curl 10.254.146.189:27017
It looks like you are trying to access MongoDB over HTTP on the native driver port.
root@mongodb:/# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:27017 0.0.0.0:* LISTEN
tcp 0 0 172.17.99.2:60724 151.101.128.204:80 TIME_WAIT
tcp 0 0 172.17.99.2:60728 151.101.128.204:80 TIME_WAIT
mongodb 容器在启动时没有错误。
尝试按照 https://kubernetes.io/docs/user-guide/debugging-services/#iptables 中的步骤进行操作，卡在“尝试将 -V 标志设置为 4 的情况下重新启动 kube-proxy”的部分，因为我不知道该怎么做。
我不是搞网络的，不知道这里面要怎么分析，分析什么。任何类型的调试技巧都会有很大帮助。
谢谢。
:)
作为旁注，请记住 curl 默认执行 HTTP 请求，但您尝试访问的主机中的端口 27017 未绑定到理解此类协议的应用程序。通常，在这些情况下您会使用 netcat：
nc -zv mongoservice 27017
报告来自该主机的端口 27017 是否打开。
- nc = 网猫
- -z 扫描侦听守护进程而不发送数据
- -v 增加冗长
关于您的 MongoDB 文件，您必须记住设置 targetPort 指令。如 Kubernetes docs regarding targetPort:
中所述
This specification will create a Service which targets TCP port 80 on any Pod with the run: my-nginx label, and expose it on an abstracted Service port (targetPort: is the port the container accepts traffic on, port: is the abstracted Service port, which can be any port other pods use to access the Service). View service API object to see the list of supported fields in service definition.
因此，为了保持一致性，只需将其设置为 27017。
遵循这些建议后，您不应 运行 陷入问题。保持良好的工作状态并尽可能多地学习！
iptables 规则看起来没问题，但不确定您的 kubernetes 使用什么网络解决方案 (flannel/calico)。您可以检查是否可以从您的 web pod 访问 kube dns pod IP。
谢谢。我对此有所了解，因为我使用的是 flannel 网络，所以 flannel 网络中 pods 之间的通信存在问题。
特别是这部分，FLANNEL_OPTIONS="--iface=eth1" 如link http://jayunit100.blogspot.com/2015/06/flannel-and-vagrant-heads-up.html
谢谢。
我有两个 pods 映射到两个服务，并且 运行 在我的笔记本电脑中使用虚拟机虚拟机。我有 kube dns 工作。一个 pod 是网络服务，另一个是 mongodb。
webapp pod 规格如下
spec:
containers:
- resources:
limits:
cpu: 0.5
.
.
name: wsemp
ports:
- containerPort: 8080
# name: wsemp
#command: ["java","-Dspring.data.mongodb.uri=mongodb://192.168.6.103:30061/microservices", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
command: ["java","-Dspring.data.mongodb.uri=mongodb://mongoservice/microservices", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
对应服务的规格
apiVersion: v1
kind: Service
metadata:
labels:
name: webappservice
name: webappservice
spec:
ports:
- port: 8080
nodePort: 30062
targetPort: 8080
protocol: TCP
type: NodePort
selector:
name: webapp
Mongodb 吊舱规格
apiVersion: v1
kind: Pod
metadata:
name: mongodb
labels:
name: mongodb
spec:
containers:
.
.
name: mongodb
ports:
- containerPort: 27017
Mongodb 服务规范
apiVersion: v1
kind: Service
metadata:
labels:
name: mongodb
name: mongoservice
spec:
ports:
- port: 27017
nodePort: 30061
targetPort: 27017
protocol: TCP
type: NodePort
selector:
name: mongodb
评论后更新的目标端口在服务中
问题
webapp 启动时无法连接到 mongoservice 端口并在启动时出现此错误
Exception in monitor thread while connecting to server mongoservice:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[mongodb-driver-core-3.2.2.jar!/:na]
at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:114) ~[mongodb-driver-core-3.2.2.jar!/:na]
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:128) ~[mongodb-driver-core-3.2.2.jar!/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_111]
描述 svc
kubectl describe svc mongoservice
Name: mongoservice
Namespace: default
Labels: name=mongodb
Selector: name=mongodb
Type: NodePort
IP: 10.254.146.189
Port: <unset> 27017/TCP
NodePort: <unset> 30061/TCP
Endpoints: 172.17.99.2:27017
Session Affinity: None
No events.
kubectl describe svc webappservice
Name: webappservice
Namespace: default
Labels: name=webappservice
Selector: name=webapp
Type: NodePort
IP: 10.254.112.121
Port: <unset> 8080/TCP
NodePort: <unset> 30062/TCP
Endpoints: 172.17.99.3:8080
Session Affinity: None
No events.
调试
root@webapp:/# nslookup mongoservice
Server: 10.254.0.2
Address: 10.254.0.2#53
Non-authoritative answer:
Name: mongoservice.default.svc.cluster.local
Address: 10.254.146.189
root@webapp:/# curl 10.254.146.189:27017
curl: (7) Failed to connect to 10.254.146.189 port 27017: Connection refused
root@webapp:/# curl mongoservice:27017
curl: (7) Failed to connect to mongoservice port 27017: Connection refused
sudo iptables-save | grep webapp
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/webappservice:" -m tcp --dport 30062 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/webappservice:" -m tcp --dport 30062 -j KUBE-SVC-NQBDRRKQULANV7O3
-A KUBE-SEP-IE7EBTQCN7T6HXC4 -s 172.17.99.3/32 -m comment --comment "default/webappservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-IE7EBTQCN7T6HXC4 -p tcp -m comment --comment "default/webappservice:" -m tcp -j DNAT --to-destination 172.17.99.3:8080
-A KUBE-SERVICES -d 10.254.217.24/32 -p tcp -m comment --comment "default/webappservice: cluster IP" -m tcp --dport 8080 -j KUBE-SVC-NQBDRRKQULANV7O3
-A KUBE-SVC-NQBDRRKQULANV7O3 -m comment --comment "default/webappservice:" -j KUBE-SEP-IE7EBTQCN7T6HXC4
$ curl 10.254.217.24:8080
{"timestamp":1486678423757,"status":404,"error":"Not Found","message":"No message available","path":"/"}[osboxes@kube-node1 ~]$
sudo iptables-save | grep mongodb
[osboxes@osboxes ~]$ sudo iptables-save | grep mongo
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/mongoservice:" -m tcp --dport 30061 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/mongoservice:" -m tcp --dport 30061 -j KUBE-SVC-2HQWGC3WSIBZF7CN
-A KUBE-SEP-FVWOWAWXXVAVIQ5O -s 172.17.99.2/32 -m comment --comment "default/mongoservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-FVWOWAWXXVAVIQ5O -p tcp -m comment --comment "default/mongoservice:" -m tcp -j DNAT --to-destination 172.17.99.2:27017
-A KUBE-SERVICES -d 10.254.146.189/32 -p tcp -m comment --comment "default/mongoservice: cluster IP" -m tcp --dport 27017 -j KUBE-SVC-2HQWGC3WSIBZF7CN
-A KUBE-SVC-2HQWGC3WSIBZF7CN -m comment --comment "default/mongoservice:" -j KUBE-SEP-FVWOWAWXXVAVIQ5O
[osboxes@osboxes ~]$ sudo curl 10.254.146.189:8080
^C[osboxes@osboxes ~]$ sudo curl 10.254.146.189:27017
It looks like you are trying to access MongoDB over HTTP on the native driver port.
root@mongodb:/# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:27017 0.0.0.0:* LISTEN
tcp 0 0 172.17.99.2:60724 151.101.128.204:80 TIME_WAIT
tcp 0 0 172.17.99.2:60728 151.101.128.204:80 TIME_WAIT
mongodb 容器在启动时没有错误。
尝试按照 https://kubernetes.io/docs/user-guide/debugging-services/#iptables 中的步骤进行操作，卡在“尝试将 -V 标志设置为 4 的情况下重新启动 kube-proxy”的部分，因为我不知道该怎么做。
我不是搞网络的，不知道这里面要怎么分析，分析什么。任何类型的调试技巧都会有很大帮助。
谢谢。
:)
作为旁注，请记住 curl 默认执行 HTTP 请求，但您尝试访问的主机中的端口 27017 未绑定到理解此类协议的应用程序。通常，在这些情况下您会使用 netcat：
nc -zv mongoservice 27017
报告来自该主机的端口 27017 是否打开。
- nc = 网猫
- -z 扫描侦听守护进程而不发送数据
- -v 增加冗长
关于您的 MongoDB 文件，您必须记住设置 targetPort 指令。如 Kubernetes docs regarding targetPort:
中所述This specification will create a Service which targets TCP port 80 on any Pod with the run: my-nginx label, and expose it on an abstracted Service port (targetPort: is the port the container accepts traffic on, port: is the abstracted Service port, which can be any port other pods use to access the Service). View service API object to see the list of supported fields in service definition.
因此，为了保持一致性，只需将其设置为 27017。
遵循这些建议后，您不应 运行 陷入问题。保持良好的工作状态并尽可能多地学习！
iptables 规则看起来没问题，但不确定您的 kubernetes 使用什么网络解决方案 (flannel/calico)。您可以检查是否可以从您的 web pod 访问 kube dns pod IP。
谢谢。我对此有所了解，因为我使用的是 flannel 网络，所以 flannel 网络中 pods 之间的通信存在问题。
特别是这部分，FLANNEL_OPTIONS="--iface=eth1" 如link http://jayunit100.blogspot.com/2015/06/flannel-and-vagrant-heads-up.html
谢谢。