在 Mesos 上 运行 k8s 时，如何通过响应错误找到匹配的 http 请求？
How can I find matched http request through a response error while running k8s on Mesos?
我参考 Getting started with Kubernetes on Mesos 并尝试在同一本地 Ubuntu 上部署 k8s 和 Mesos。执行以下命令：
$ km controller-manager \
--master=${KUBERNETES_MASTER_IP}:8888 \
--cloud-provider=mesos \
--cloud-config=./mesos-cloud.conf \
--v=8 >controller.log 2>&1 &
来自controller.log：
$ more controller.log
I1210 05:33:13.570789 12082 controllermanager.go:217] Creating hostIP:hostPort endpoint controller
I1210 05:33:13.571253 12082 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1210 05:33:13.571294 12082 standalone.go:46] creating new standalone detector for &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16
.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.571469 12082 standalone.go:67] Detect()
I1210 05:33:13.571485 12082 standalone.go:69] spinning up asyc master detector poller
I1210 05:33:13.571501 12082 standalone.go:78] spawning asyc master detector listener
I1210 05:33:13.571608 12082 nodecontroller.go:132] Sending events to api server.
I1210 05:33:13.571706 12082 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1210 05:33:13.571729 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/services
I1210 05:33:13.571757 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.571772 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.571831 12082 standalone.go:80] waiting for polled to send updates
I1210 05:33:13.571986 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/replicationcontrollers
I1210 05:33:13.572010 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572026 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
E1210 05:33:13.571994 12082 controllermanager.go:152] Failed to start service controller: the cloud provider does not support TCP load ba
lancers.
I1210 05:33:13.572124 12082 round_trippers.go:222] GET http://16.187.250.141:8888/apis/extensions/v1beta1/daemonsets
I1210 05:33:13.572151 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572167 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572208 12082 persistentvolume_claim_binder_controller.go:346] Starting PersistentVolumeClaimBinder
I1210 05:33:13.572262 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572268 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572300 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572314 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572305 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572331 12082 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1210 05:33:13.572371 12082 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1210 05:33:13.572393 12082 persistentvolume_recycler_controller.go:211] Starting PersistentVolumeRecycler
I1210 05:33:13.572452 12082 resource_quota_controller.go:133] Resource quota controller queued all resource quota for full calculation of
usage
I1210 05:33:13.572479 12082 client.go:77] Reloading cached Mesos state
I1210 05:33:13.572283 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572499 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572346 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572508 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572531 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572535 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572547 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572571 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572584 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572521 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572612 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572632 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.572662 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572676 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572684 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/resourcequotas
I1210 05:33:13.572702 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572714 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572756 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumeclaims
I1210 05:33:13.572551 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572791 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572815 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572329 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572846 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572858 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572864 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/serviceaccounts?fieldSelector=metadata.name%3Dde
fault
I1210 05:33:13.572869 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572885 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572892 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572777 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572948 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572167 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573010 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573019 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572834 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573072 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572876 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573120 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572641 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.573193 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573202 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.573286 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573304 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573313 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.576806 12082 standalone.go:214] Got mesos state, content length 1795
I1210 05:33:13.577013 12082 standalone.go:147] detected leadership change from '<nil>' to 'master@16.187.250.141:5050'
I1210 05:33:13.577098 12082 standalone.go:185] master leader poller sleeping for 29.994633799s
I1210 05:33:13.577118 12082 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:
5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.577218 12082 client.go:155] cloud master changed to '16.187.250.141:5050'
I1210 05:33:14.192579 12082 round_trippers.go:247] Response Status: 403 Forbidden in 619 milliseconds
I1210 05:33:14.192610 12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192621 12082 round_trippers.go:253] Proxy-Connection: Keep-Alive
E1210 05:33:14.192639 12082 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
I1210 05:33:14.192667 12082 round_trippers.go:253] Connection: Keep-Alive
I1210 05:33:14.192692 12082 round_trippers.go:253] Content-Length: 606
I1210 05:33:14.192703 12082 round_trippers.go:253] Cache-Control: no-cache
I1210 05:33:14.192712 12082 round_trippers.go:253] Pragma: no-cache
I1210 05:33:14.192722 12082 round_trippers.go:253] Content-Type: text/html; charset=utf-8
I1210 05:33:14.192765 12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.192800 12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192812 12082 round_trippers.go:253] Pragma: no-cache
I1210 05:33:14.192823 12082 round_trippers.go:253] Content-Type: text/html; charset=utf-8
I1210 05:33:14.192833 12082 round_trippers.go:253] Proxy-Connection: Keep-Alive
I1210 05:33:14.192842 12082 round_trippers.go:253] Connection: Keep-Alive
I1210 05:33:14.192851 12082 round_trippers.go:253] Content-Length: 606
I1210 05:33:14.192861 12082 round_trippers.go:253] Cache-Control: no-cache
I1210 05:33:14.192905 12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192935 12082 request.go:861] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192785 12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.193025 12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.193034 12082 request.go:861] Response Body: <HTML><HEAD>
我可以看到有一些“403”http 响应错误。但是如何找到导致此响应错误的 http 请求？
经过调查，分两步：
(1) 使用 lsof 找到 http 连接：
lsof -p pid -P -n
(2)使用tcpdump抓取网络包：
sudo tcpdump -vv -A -s 0 'tcp port xxxx' -i em1 -w capture.pcap
然后分析包。
我参考 Getting started with Kubernetes on Mesos 并尝试在同一本地 Ubuntu 上部署 k8s 和 Mesos。执行以下命令：
$ km controller-manager \
--master=${KUBERNETES_MASTER_IP}:8888 \
--cloud-provider=mesos \
--cloud-config=./mesos-cloud.conf \
--v=8 >controller.log 2>&1 &
来自controller.log：
$ more controller.log
I1210 05:33:13.570789 12082 controllermanager.go:217] Creating hostIP:hostPort endpoint controller
I1210 05:33:13.571253 12082 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1210 05:33:13.571294 12082 standalone.go:46] creating new standalone detector for &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16
.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.571469 12082 standalone.go:67] Detect()
I1210 05:33:13.571485 12082 standalone.go:69] spinning up asyc master detector poller
I1210 05:33:13.571501 12082 standalone.go:78] spawning asyc master detector listener
I1210 05:33:13.571608 12082 nodecontroller.go:132] Sending events to api server.
I1210 05:33:13.571706 12082 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1210 05:33:13.571729 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/services
I1210 05:33:13.571757 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.571772 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.571831 12082 standalone.go:80] waiting for polled to send updates
I1210 05:33:13.571986 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/replicationcontrollers
I1210 05:33:13.572010 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572026 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
E1210 05:33:13.571994 12082 controllermanager.go:152] Failed to start service controller: the cloud provider does not support TCP load ba
lancers.
I1210 05:33:13.572124 12082 round_trippers.go:222] GET http://16.187.250.141:8888/apis/extensions/v1beta1/daemonsets
I1210 05:33:13.572151 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572167 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572208 12082 persistentvolume_claim_binder_controller.go:346] Starting PersistentVolumeClaimBinder
I1210 05:33:13.572262 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572268 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572300 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572314 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572305 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572331 12082 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1210 05:33:13.572371 12082 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1210 05:33:13.572393 12082 persistentvolume_recycler_controller.go:211] Starting PersistentVolumeRecycler
I1210 05:33:13.572452 12082 resource_quota_controller.go:133] Resource quota controller queued all resource quota for full calculation of
usage
I1210 05:33:13.572479 12082 client.go:77] Reloading cached Mesos state
I1210 05:33:13.572283 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572499 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572346 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572508 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572531 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572535 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572547 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572571 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572584 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572521 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572612 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572632 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.572662 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572676 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572684 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/resourcequotas
I1210 05:33:13.572702 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572714 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572756 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumeclaims
I1210 05:33:13.572551 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572791 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572815 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572329 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572846 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572858 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572864 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/serviceaccounts?fieldSelector=metadata.name%3Dde
fault
I1210 05:33:13.572869 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572885 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572892 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572777 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572948 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572167 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573010 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573019 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572834 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573072 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572876 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573120 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572641 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.573193 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573202 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.573286 12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573304 12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573313 12082 round_trippers.go:232] User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.576806 12082 standalone.go:214] Got mesos state, content length 1795
I1210 05:33:13.577013 12082 standalone.go:147] detected leadership change from '<nil>' to 'master@16.187.250.141:5050'
I1210 05:33:13.577098 12082 standalone.go:185] master leader poller sleeping for 29.994633799s
I1210 05:33:13.577118 12082 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:
5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.577218 12082 client.go:155] cloud master changed to '16.187.250.141:5050'
I1210 05:33:14.192579 12082 round_trippers.go:247] Response Status: 403 Forbidden in 619 milliseconds
I1210 05:33:14.192610 12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192621 12082 round_trippers.go:253] Proxy-Connection: Keep-Alive
E1210 05:33:14.192639 12082 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
I1210 05:33:14.192667 12082 round_trippers.go:253] Connection: Keep-Alive
I1210 05:33:14.192692 12082 round_trippers.go:253] Content-Length: 606
I1210 05:33:14.192703 12082 round_trippers.go:253] Cache-Control: no-cache
I1210 05:33:14.192712 12082 round_trippers.go:253] Pragma: no-cache
I1210 05:33:14.192722 12082 round_trippers.go:253] Content-Type: text/html; charset=utf-8
I1210 05:33:14.192765 12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.192800 12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192812 12082 round_trippers.go:253] Pragma: no-cache
I1210 05:33:14.192823 12082 round_trippers.go:253] Content-Type: text/html; charset=utf-8
I1210 05:33:14.192833 12082 round_trippers.go:253] Proxy-Connection: Keep-Alive
I1210 05:33:14.192842 12082 round_trippers.go:253] Connection: Keep-Alive
I1210 05:33:14.192851 12082 round_trippers.go:253] Content-Length: 606
I1210 05:33:14.192861 12082 round_trippers.go:253] Cache-Control: no-cache
I1210 05:33:14.192905 12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192935 12082 request.go:861] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192785 12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.193025 12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.193034 12082 request.go:861] Response Body: <HTML><HEAD>
我可以看到有一些“403”http 响应错误。但是如何找到导致此响应错误的 http 请求？
经过调查，分两步：
(1) 使用 lsof 找到 http 连接：
lsof -p pid -P -n
(2)使用tcpdump抓取网络包：
sudo tcpdump -vv -A -s 0 'tcp port xxxx' -i em1 -w capture.pcap
然后分析包。