state_replicaset/state_replicaset.go98 发出 http 请求时出错：Get kube-state-metrics:8080/metrics: lookup kube-state-metrics on IP:53 no such host

Question

我们正尝试在台风 kubernetes 集群上启动 metricbeat。但是在启动之后，由于以下原因，它无法获得一些特定于 pod 的事件，例如重启等

对应的metricbeat.yaml片段

# State metrics from kube-state-metrics service:
    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_statefulset
        - state_pod
        - state_container
        - state_cronjob
        - state_resourcequota
        - state_service
        - state_persistentvolume
        - state_persistentvolumeclaim
        - state_storageclass
        # Uncomment this to get k8s events:
        #- event  period: 10s
      hosts: ["kube-state-metrics:8080"]

我们面临的错误

2020-07-01T10:31:02.486Z        ERROR   [kubernetes.state_statefulset]  state_statefulset/state_statefulset.go:97       error making http request: Get http://kube-state-metrics:8080/metrics: lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:02.611Z        WARN    [transport]     transport/tcp.go:52     DNS lookup failure "kube-state-metrics": lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:02.611Z        INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.state_node: error doing HTTP request to fetch 'state_node' Metricset data: error making http request: Get http://kube-state-metrics:8080/metrics: lookup kube-state-metrics on *.*.*.*:53: no such host
2020-07-01T10:31:03.313Z        ERROR   process_summary/process_summary.go:102  Unknown or unexpected state <P> for process with pid 19
2020-07-01T10:31:03.313Z        ERROR   process_summary/process_summary.go:102  Unknown or unexpected state <P> for process with pid 20

我可以添加一些其他需要的信息。

Answer 1

确保您在集群中的 kube-system 命名空间中部署了 Kube-State-Metrics 以使其正常工作。 Metricbeat 默认不会自带这个。

请参阅this了解详细的部署说明。

Answer 2

如果您的 kube-state-metrics 部署到另一个名称空间，Kubernetes 无法解析该名称。例如。我们已将 kube-state-metrics 部署到 monitoring 命名空间：

$ kubectl get pods -A | grep kube-state-metrics
monitoring             kube-state-metrics-765c7c7f95-v7mmp                             3/3     Running   17         10d

您可以将 hosts 选项设置为全名，包括命名空间，如下所示：

    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_statefulset
        - state_pod
        - state_container
        - state_cronjob
        - state_resourcequota
        - state_service
        - state_persistentvolume
        - state_persistentvolumeclaim
        - state_storageclass
      hosts: ["kube-state-metrics.<your_namespace>:8080"]

state_replicaset/state_replicaset.go98 发出 http 请求时出错：Get kube-state-metrics:8080/metrics: lookup kube-state-metrics on IP:53 no such host

state_replicaset/state_replicaset.go98 error making http request: Get kube-state-metrics:8080/metrics: lookup kube-state-metrics on IP:53 no such host

elasticsearch

kubernetes

metricbeat

typhoon-kubernetes

对应的metricbeat.yaml片段

我们面临的错误