Kubernetes 端点抛出“ServiceUnavailable”
Kubernetes endpoints throw `ServiceUnavailable`
我在 AWS 上有一个新的 Kubernetes 集群,它是使用来自 v1.1.1 的 kube-up
脚本构建的。我可以成功访问 Elasticsearch/Kibana/KubeUI/Grafana 端点,但无法通过 API 代理从我的机器访问 Heapster/KubeDNS/InfluxDB。我在 K8S 项目上看到了一些与此相关的辅助问题,但没有明确确定发生了什么。据我所知,一切 运行 都很好,所以我不确定这里出了什么问题?我真的很想使用 Grafana/Influx/Heapster 的嵌入式监控,但 Grafana 仪表板只是一片空白,并出现一系列错误。
Kubernetes 版本
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.1", GitCommit:"92635e23dfafb2ddc828c8ac6c03c7a7205a84d8", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.1", GitCommit:"92635e23dfafb2ddc828c8ac6c03c7a7205a84d8", GitTreeState:"clean"}
集群信息
$ kubectl cluster-info
Kubernetes master is running at https://MASTER_IP
Elasticsearch is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
Heapster is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/heapster
Kibana is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/kibana-logging
KubeDNS is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/kube-dns
KubeUI is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/kube-ui
Grafana is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
从上面的 API 代理 URL 访问 influxDB
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service \"monitoring-influxdb\"",
"reason": "ServiceUnavailable",
"code": 503
}
来自主机的端点详细信息
$ curl http://localhost:8080/api/v1/namespaces/kube-system/endpoints/monitoring-influxdb
{
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "monitoring-influxdb",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/endpoints/monitoring-influxdb",
"uid": "2f75b259-8a22-11e5-b248-028ff74b9b1b",
"resourceVersion": "131",
"creationTimestamp": "2015-11-13T16:18:33Z",
"labels": {
"kubernetes.io/cluster-service": "true",
"kubernetes.io/name": "InfluxDB"
}
},
"subsets": [
{
"addresses": [
{
"ip": "10.244.1.4",
"targetRef": {
"kind": "Pod",
"namespace": "kube-system",
"name": "monitoring-influxdb-grafana-v2-n6jx1",
"uid": "2f31ed90-8a22-11e5-b248-028ff74b9b1b",
"resourceVersion": "127"
}
}
],
"ports": [
{
"name": "http",
"port": 8083,
"protocol": "TCP"
},
{
"name": "api",
"port": 8086,
"protocol": "TCP"
}
]
}
]
}
正在从主机查询服务
$ curl -IL 10.244.1.4:8083
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 13751
Content-Type: text/html; charset=utf-8
Last-Modified: Fri, 14 Nov 2014 21:55:58 GMT
Date: Tue, 17 Nov 2015 21:31:48 GMT
监控-InfluxDB 服务
$ curl http://localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "monitoring-influxdb",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/services/monitoring-influxdb",
"uid": "2f715831-8a22-11e5-b248-028ff74b9b1b",
"resourceVersion": "60",
"creationTimestamp": "2015-11-13T16:18:33Z",
"labels": {
"kubernetes.io/cluster-service": "true",
"kubernetes.io/name": "InfluxDB"
}
},
"spec": {
"ports": [
{
"name": "http",
"protocol": "TCP",
"port": 8083,
"targetPort": 8083
},
{
"name": "api",
"protocol": "TCP",
"port": 8086,
"targetPort": 8086
}
],
"selector": {
"k8s-app": "influxGrafana"
},
"clusterIP": "10.0.35.241",
"type": "ClusterIP",
"sessionAffinity": "None"
},
"status": {
"loadBalancer": {}
}
}
连播详情
$ kubectl describe pod --namespace=kube-system monitoring-influxdb-grafana-v2-n6jx
Name: monitoring-influxdb-grafana-v2-n6jx1
Namespace: kube-system
Image(s): gcr.io/google_containers/heapster_influxdb:v0.4,beta.gcr.io/google_containers/heapster_grafana:v2.1.1
Node: ip-172-20-0-44.us-west-2.compute.internal/172.20.0.44
Start Time: Fri, 13 Nov 2015 08:21:36 -0800
Labels: k8s-app=influxGrafana,kubernetes.io/cluster-service=true,version=v2
Status: Running
Reason:
Message:
IP: 10.244.1.4
Replication Controllers: monitoring-influxdb-grafana-v2 (1/1 replicas created)
Containers:
influxdb:
Container ID: docker://564724318ca81d33d6079978d24f78b3c6ff8eb08a9023c845e250eeb888aafd
Image: gcr.io/google_containers/heapster_influxdb:v0.4
Image ID: docker://8b8118c488e431cc43e7ff9060968d88402cc6c38a6390c4221352403aa7ac1b
QoS Tier:
memory: Guaranteed
cpu: Guaranteed
Limits:
memory: 200Mi
cpu: 100m
Requests:
memory: 200Mi
cpu: 100m
State: Running
Started: Fri, 13 Nov 2015 08:22:55 -0800
Ready: True
Restart Count: 0
Environment Variables:
grafana:
Container ID: docker://518dea564a0ee014345e9006da6113fb6584ff1ebc6d0cc9609a608abc995f45
Image: beta.gcr.io/google_containers/heapster_grafana:v2.1.1
Image ID: docker://200e77ba156a5a86879e49667b97afe84dca42b5bb67ab1e06217e6a19c5a6a6
QoS Tier:
cpu: Guaranteed
memory: Guaranteed
Limits:
memory: 100Mi
cpu: 100m
Requests:
cpu: 100m
memory: 100Mi
State: Running
Started: Fri, 13 Nov 2015 08:22:35 -0800
Ready: True
Restart Count: 0
Environment Variables:
INFLUXDB_SERVICE_URL: http://monitoring-influxdb:8086
GF_AUTH_BASIC_ENABLED: false
GF_AUTH_ANONYMOUS_ENABLED: true
GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
GF_SERVER_ROOT_URL: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/
Conditions:
Type Status
Ready True
Volumes:
influxdb-persistent-storage:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
grafana-persistent-storage:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-bo89c:
Type: Secret (a secret that should populate this volume)
SecretName: default-token-bo89c
No events.
遗憾的是,这些 URL 不完整。 Influx的端口是有名字的,所以你需要说出你想要哪个端口。
https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:http
或 https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:api
有一个 bug 可以为此提供更好的错误。
我在 AWS 上有一个新的 Kubernetes 集群,它是使用来自 v1.1.1 的 kube-up
脚本构建的。我可以成功访问 Elasticsearch/Kibana/KubeUI/Grafana 端点,但无法通过 API 代理从我的机器访问 Heapster/KubeDNS/InfluxDB。我在 K8S 项目上看到了一些与此相关的辅助问题,但没有明确确定发生了什么。据我所知,一切 运行 都很好,所以我不确定这里出了什么问题?我真的很想使用 Grafana/Influx/Heapster 的嵌入式监控,但 Grafana 仪表板只是一片空白,并出现一系列错误。
Kubernetes 版本
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.1", GitCommit:"92635e23dfafb2ddc828c8ac6c03c7a7205a84d8", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.1", GitCommit:"92635e23dfafb2ddc828c8ac6c03c7a7205a84d8", GitTreeState:"clean"}
集群信息
$ kubectl cluster-info
Kubernetes master is running at https://MASTER_IP
Elasticsearch is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
Heapster is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/heapster
Kibana is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/kibana-logging
KubeDNS is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/kube-dns
KubeUI is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/kube-ui
Grafana is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
从上面的 API 代理 URL 访问 influxDB
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service \"monitoring-influxdb\"",
"reason": "ServiceUnavailable",
"code": 503
}
来自主机的端点详细信息
$ curl http://localhost:8080/api/v1/namespaces/kube-system/endpoints/monitoring-influxdb
{
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "monitoring-influxdb",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/endpoints/monitoring-influxdb",
"uid": "2f75b259-8a22-11e5-b248-028ff74b9b1b",
"resourceVersion": "131",
"creationTimestamp": "2015-11-13T16:18:33Z",
"labels": {
"kubernetes.io/cluster-service": "true",
"kubernetes.io/name": "InfluxDB"
}
},
"subsets": [
{
"addresses": [
{
"ip": "10.244.1.4",
"targetRef": {
"kind": "Pod",
"namespace": "kube-system",
"name": "monitoring-influxdb-grafana-v2-n6jx1",
"uid": "2f31ed90-8a22-11e5-b248-028ff74b9b1b",
"resourceVersion": "127"
}
}
],
"ports": [
{
"name": "http",
"port": 8083,
"protocol": "TCP"
},
{
"name": "api",
"port": 8086,
"protocol": "TCP"
}
]
}
]
}
正在从主机查询服务
$ curl -IL 10.244.1.4:8083
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 13751
Content-Type: text/html; charset=utf-8
Last-Modified: Fri, 14 Nov 2014 21:55:58 GMT
Date: Tue, 17 Nov 2015 21:31:48 GMT
监控-InfluxDB 服务
$ curl http://localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "monitoring-influxdb",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/services/monitoring-influxdb",
"uid": "2f715831-8a22-11e5-b248-028ff74b9b1b",
"resourceVersion": "60",
"creationTimestamp": "2015-11-13T16:18:33Z",
"labels": {
"kubernetes.io/cluster-service": "true",
"kubernetes.io/name": "InfluxDB"
}
},
"spec": {
"ports": [
{
"name": "http",
"protocol": "TCP",
"port": 8083,
"targetPort": 8083
},
{
"name": "api",
"protocol": "TCP",
"port": 8086,
"targetPort": 8086
}
],
"selector": {
"k8s-app": "influxGrafana"
},
"clusterIP": "10.0.35.241",
"type": "ClusterIP",
"sessionAffinity": "None"
},
"status": {
"loadBalancer": {}
}
}
连播详情
$ kubectl describe pod --namespace=kube-system monitoring-influxdb-grafana-v2-n6jx
Name: monitoring-influxdb-grafana-v2-n6jx1
Namespace: kube-system
Image(s): gcr.io/google_containers/heapster_influxdb:v0.4,beta.gcr.io/google_containers/heapster_grafana:v2.1.1
Node: ip-172-20-0-44.us-west-2.compute.internal/172.20.0.44
Start Time: Fri, 13 Nov 2015 08:21:36 -0800
Labels: k8s-app=influxGrafana,kubernetes.io/cluster-service=true,version=v2
Status: Running
Reason:
Message:
IP: 10.244.1.4
Replication Controllers: monitoring-influxdb-grafana-v2 (1/1 replicas created)
Containers:
influxdb:
Container ID: docker://564724318ca81d33d6079978d24f78b3c6ff8eb08a9023c845e250eeb888aafd
Image: gcr.io/google_containers/heapster_influxdb:v0.4
Image ID: docker://8b8118c488e431cc43e7ff9060968d88402cc6c38a6390c4221352403aa7ac1b
QoS Tier:
memory: Guaranteed
cpu: Guaranteed
Limits:
memory: 200Mi
cpu: 100m
Requests:
memory: 200Mi
cpu: 100m
State: Running
Started: Fri, 13 Nov 2015 08:22:55 -0800
Ready: True
Restart Count: 0
Environment Variables:
grafana:
Container ID: docker://518dea564a0ee014345e9006da6113fb6584ff1ebc6d0cc9609a608abc995f45
Image: beta.gcr.io/google_containers/heapster_grafana:v2.1.1
Image ID: docker://200e77ba156a5a86879e49667b97afe84dca42b5bb67ab1e06217e6a19c5a6a6
QoS Tier:
cpu: Guaranteed
memory: Guaranteed
Limits:
memory: 100Mi
cpu: 100m
Requests:
cpu: 100m
memory: 100Mi
State: Running
Started: Fri, 13 Nov 2015 08:22:35 -0800
Ready: True
Restart Count: 0
Environment Variables:
INFLUXDB_SERVICE_URL: http://monitoring-influxdb:8086
GF_AUTH_BASIC_ENABLED: false
GF_AUTH_ANONYMOUS_ENABLED: true
GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
GF_SERVER_ROOT_URL: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/
Conditions:
Type Status
Ready True
Volumes:
influxdb-persistent-storage:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
grafana-persistent-storage:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-bo89c:
Type: Secret (a secret that should populate this volume)
SecretName: default-token-bo89c
No events.
遗憾的是,这些 URL 不完整。 Influx的端口是有名字的,所以你需要说出你想要哪个端口。
https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:http
或 https://MASTER_IP/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:api
有一个 bug 可以为此提供更好的错误。