Azure k8s windows cluster, Internal Error: Missing stats for root container
Azure k8s windows cluster, Internal Error: Missing stats for root container
我创建了一个 k8s windows 集群（1 个主节点和 1 个节点）。 kubernetes 仪表板显示了两个节点。仪表板仅显示 master 的资源利用率。
从仪表板检查 heapster 日志时，我看到以下错误：
... E0331 18:17:05.029078 1 summary.go:97] error while getting metrics summary from Kubelet a67a6acs9000(10.240.0.4:10255): request failed - "500 Internal Server Error", response: "Internal Error: Missing stats for root container"
Kubulet在windows节点上的日志出现如下错误：
E0331 16:53:48.527868 900 eviction_manager.go:205] eviction manager: unexpected err: Missing stats for root container
E0331 16:53:55.757082 900 kubelet.go:1634] Failed to check if disk space is available for the runtime: could not determine capacity for "runtime" fs. Info: {Usage:0 Capacity:0 Available:0 Timestamp:2017-03-31 16:53:55.7570827 +0000 GMT}
E0331 16:53:55.757082 900 kubelet.go:1642] Failed to check if disk space is available on the root partition: could not determine capacity for "root" fs. Info: {Usage:0 Capacity:0 Available:0 Timestamp:2017-03-31 16:53:55.7570827 +0000 G
E0331 16:54:04.462246 900 handler.go:246] HTTP InternalServerError serving /stats/summary: Internal Error: Missing stats for root container
I0331 16:54:04.463246 900 server.go:740] GET /stats/summary/: (5.006ms) 500
goroutine 920818 [running]:
k8s.io/kubernetes/pkg/httplog.(*respLogger).recordStatus(0xc042334a10, 0x1f4)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xc2
k8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc042334a10, 0x1f4)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x3c
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteHeader(0xc042224a80, 0x1f4)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:200 +0x48
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteErrorString(0xc042224a80, 0x1f4, 0xc043082780, 0x30, 0x2, 0xc043082780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:180 +0x4d
k8s.io/kubernetes/pkg/kubelet/server/stats.handleError(0xc042224a80, 0x2893ade, 0xe, 0x3aa6880, 0xc042448830)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:247 +0x1fe
k8s.io/kubernetes/pkg/kubelet/server/stats.(*handler).handleSummary(0xc0427e9e80, 0xc042bfa5d0, 0xc042224a80)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:158 +0x80
k8s.io/kubernetes/pkg/kubelet/server/stats.(*handler).(k8s.io/kubernetes/pkg/kubelet/server/stats.handleSummary)-fm(0xc042bfa5d0, 0xc042224a80)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:70 +0x45
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc04235fc20, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xba7
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x4f
net/http.HandlerFunc.ServeHTTP(0xc0422c05c0, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/go/src/net/http/server.go:1726 +0x4b
net/http.(*ServeMux).ServeHTTP(0xc0427d88d0, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/go/src/net/http/server.go:2022 +0x86
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP(0xc04235fc20, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:287 +0x54
k8s.io/kubernetes/pkg/kubelet/server.(*Server).ServeHTTP(0xc0427e2690, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:739 +0x115
net/http.serverHandler.ServeHTTP(0xc0421dcf80, 0x3ac33c0, 0xc04234de10, 0xc042a10780)
/home/jiangtianli/go/src/net/http/server.go:2202 +0x84
net/http.(*conn).serve(0xc042c0dd00, 0x3ac51c0, 0xc042ab3f00)
/home/jiangtianli/go/src/net/http/server.go:1579 +0x4be
created by net/http.(*Server).Serve
/home/jiangtianli/go/src/net/http/server.go:2293 +0x454
ACS 上的默认 k8s 集群设置使用 Heapster v1.2.0，我注意到更新版本 v1.3.0 可用。所以我升级了heapster，但是错误依然存在
感谢您的报告。 ACS Windows Kubernetes 仍处于早期开发阶段，需要清理所有常见问题。我在 Kubernetes 上提交了以下任务来跟踪这个：https://github.com/kubernetes/kubernetes/issues/43993.
我创建了一个 k8s windows 集群（1 个主节点和 1 个节点）。 kubernetes 仪表板显示了两个节点。仪表板仅显示 master 的资源利用率。
从仪表板检查 heapster 日志时，我看到以下错误：
... E0331 18:17:05.029078 1 summary.go:97] error while getting metrics summary from Kubelet a67a6acs9000(10.240.0.4:10255): request failed - "500 Internal Server Error", response: "Internal Error: Missing stats for root container"
Kubulet在windows节点上的日志出现如下错误：
E0331 16:53:48.527868 900 eviction_manager.go:205] eviction manager: unexpected err: Missing stats for root container
E0331 16:53:55.757082 900 kubelet.go:1634] Failed to check if disk space is available for the runtime: could not determine capacity for "runtime" fs. Info: {Usage:0 Capacity:0 Available:0 Timestamp:2017-03-31 16:53:55.7570827 +0000 GMT}
E0331 16:53:55.757082 900 kubelet.go:1642] Failed to check if disk space is available on the root partition: could not determine capacity for "root" fs. Info: {Usage:0 Capacity:0 Available:0 Timestamp:2017-03-31 16:53:55.7570827 +0000 G
E0331 16:54:04.462246 900 handler.go:246] HTTP InternalServerError serving /stats/summary: Internal Error: Missing stats for root container
I0331 16:54:04.463246 900 server.go:740] GET /stats/summary/: (5.006ms) 500
goroutine 920818 [running]:
k8s.io/kubernetes/pkg/httplog.(*respLogger).recordStatus(0xc042334a10, 0x1f4)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xc2
k8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc042334a10, 0x1f4)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x3c
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteHeader(0xc042224a80, 0x1f4)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:200 +0x48
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteErrorString(0xc042224a80, 0x1f4, 0xc043082780, 0x30, 0x2, 0xc043082780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:180 +0x4d
k8s.io/kubernetes/pkg/kubelet/server/stats.handleError(0xc042224a80, 0x2893ade, 0xe, 0x3aa6880, 0xc042448830)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:247 +0x1fe
k8s.io/kubernetes/pkg/kubelet/server/stats.(*handler).handleSummary(0xc0427e9e80, 0xc042bfa5d0, 0xc042224a80)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:158 +0x80
k8s.io/kubernetes/pkg/kubelet/server/stats.(*handler).(k8s.io/kubernetes/pkg/kubelet/server/stats.handleSummary)-fm(0xc042bfa5d0, 0xc042224a80)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:70 +0x45
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc04235fc20, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xba7
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x4f
net/http.HandlerFunc.ServeHTTP(0xc0422c05c0, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/go/src/net/http/server.go:1726 +0x4b
net/http.(*ServeMux).ServeHTTP(0xc0427d88d0, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/go/src/net/http/server.go:2022 +0x86
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP(0xc04235fc20, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:287 +0x54
k8s.io/kubernetes/pkg/kubelet/server.(*Server).ServeHTTP(0xc0427e2690, 0x3abfe00, 0xc042334a10, 0xc042a10780)
/home/jiangtianli/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:739 +0x115
net/http.serverHandler.ServeHTTP(0xc0421dcf80, 0x3ac33c0, 0xc04234de10, 0xc042a10780)
/home/jiangtianli/go/src/net/http/server.go:2202 +0x84
net/http.(*conn).serve(0xc042c0dd00, 0x3ac51c0, 0xc042ab3f00)
/home/jiangtianli/go/src/net/http/server.go:1579 +0x4be
created by net/http.(*Server).Serve
/home/jiangtianli/go/src/net/http/server.go:2293 +0x454
ACS 上的默认 k8s 集群设置使用 Heapster v1.2.0，我注意到更新版本 v1.3.0 可用。所以我升级了heapster，但是错误依然存在
感谢您的报告。 ACS Windows Kubernetes 仍处于早期开发阶段，需要清理所有常见问题。我在 Kubernetes 上提交了以下任务来跟踪这个：https://github.com/kubernetes/kubernetes/issues/43993.