K8S HPA - 无法从外部指标中获取指标 API
K8S HPA - Cannot fetch metrics from External metrics API
我正在尝试将 Kafka 主题滞后到 Prometheus，最后到 API服务器，以便为我的应用程序使用外部指标 HPA。
我收到错误 没有从外部指标返回指标 API
70m Warning FailedGetExternalMetric horizontalpodautoscaler/kafkademo-hpa unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kafkademo-hpa invalid metrics (1 invalid out of 1), first error is: failed to get external metric kafka_lag_metric_sm0ke: unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
即使我在查询外部 API:
时可以看到以下输出，也会发生这种情况
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "kafka_lag_metric_sm0ke",
"singularName": "",
"namespaced": true,
"kind": "ExternalMetricValueList",
"verbs": [
"get"
]
}
]
}
设置如下：
- 卡夫卡：v2.7.0
- 普罗米修斯：v2.26.0
- 普罗米修斯适配器：v0.8.3
Prometheus 适配器值
rules:
external:
- seriesQuery: 'kafka_consumergroup_group_lag{topic="prices"}'
resources:
template: <<.Resource>>
name:
as: "kafka_lag_metric_sm0ke"
metricsQuery: 'avg by (topic) (round(avg_over_time(<<.Series>>{<<.LabelMatchers>>}[1m])))'
HPA
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: kafkademo-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kafkademo
minReplicas: 3
maxReplicas: 12
metrics:
- type: External
external:
metricName: kafka_lag_metric_sm0ke
metricSelector:
matchLabels:
topic: prices
targetValue: 5
HPA 信息
kubectl describe hpa kafkademo-hpa
Name: kafkademo-hpa
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Sat, 17 Apr 2021 20:01:29 +0300
Reference: Deployment/kafkademo
Metrics: ( current / target )
"kafka_lag_metric_sm0ke" (target value): <unknown> / 5
Min replicas: 3
Max replicas: 12
Deployment pods: 3 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 70m (x335 over 155m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get external metric kafka_lag_metric_sm0ke: unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
Warning FailedGetExternalMetric 2m30s (x366 over 155m) horizontal-pod-autoscaler unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
-- 编辑1
当我查询默认命名空间时，我得到这个：
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/kafka_lag_metric_sm0ke |jq
{
"kind": "ExternalMetricValueList",
"apiVersion": "external.metrics.k8s.io/v1beta1",
"metadata": {},
"items": []
}
我可以看到“项目”字段是空的。这是什么意思？
我似乎不理解幕后发生的一系列事件。
据我所知，这就是发生的事情。 这是正确的吗？
- prometheus-adapter 查询 Prometheus，执行 seriesQuery，计算 metricsQuery 并创建“kafka_lag_metric_sm0ke”
- 它向 api 服务器注册一个端点以获取外部指标。
- API 服务器将根据该端点定期更新其统计信息。
- HPA 从 API 服务器检查“kafka_lag_metric_sm0ke”并根据提供的值执行缩放。
我似乎也不明白名称空间在这一切中的重要性。我可以看到该统计信息已命名空间。这是否意味着每个命名空间将有 1 个统计信息？这有什么意义？
在我提出问题后回答我自己的问题的悠久传统，以上配置有什么问题。
错误在prometheus-adapter yaml:
rules:
external:
- seriesQuery: 'kafka_consumergroup_group_lag{topic="prices"}'
resources:
template: <<.Resource>>
name:
as: "kafka_lag_metric_sm0ke"
metricsQuery: 'avg by (topic) (round(avg_over_time(<<.Series>>{<<.LabelMatchers>>}[1m])))'
我删除了 <<.LabelMatchers>>，现在可以使用了：
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/kafka_lag_metric_sm0ke |jq
{
"kind": "ExternalMetricValueList",
"apiVersion": "external.metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metricName": "kafka_lag_metric_sm0ke",
"metricLabels": {
"topic": "prices"
},
"timestamp": "2021-04-21T16:55:18Z",
"value": "0"
}
]
}
我仍然不确定它为什么有效。我知道 <<.LabelMatchers>> 在这种情况下将被替换为不会产生有效查询的内容，但我不知道它是什么。
@sm0ke21
不支持 hpa api 版本 autoscaling/v2beta1，您应该使用 autoscaling/v2beta2 或新的 api 版本
我正在尝试将 Kafka 主题滞后到 Prometheus，最后到 API服务器，以便为我的应用程序使用外部指标 HPA。
我收到错误 没有从外部指标返回指标 API
70m Warning FailedGetExternalMetric horizontalpodautoscaler/kafkademo-hpa unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kafkademo-hpa invalid metrics (1 invalid out of 1), first error is: failed to get external metric kafka_lag_metric_sm0ke: unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
即使我在查询外部 API:
时可以看到以下输出，也会发生这种情况kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "kafka_lag_metric_sm0ke",
"singularName": "",
"namespaced": true,
"kind": "ExternalMetricValueList",
"verbs": [
"get"
]
}
]
}
设置如下：
- 卡夫卡：v2.7.0
- 普罗米修斯：v2.26.0
- 普罗米修斯适配器：v0.8.3
Prometheus 适配器值
rules:
external:
- seriesQuery: 'kafka_consumergroup_group_lag{topic="prices"}'
resources:
template: <<.Resource>>
name:
as: "kafka_lag_metric_sm0ke"
metricsQuery: 'avg by (topic) (round(avg_over_time(<<.Series>>{<<.LabelMatchers>>}[1m])))'
HPA
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: kafkademo-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kafkademo
minReplicas: 3
maxReplicas: 12
metrics:
- type: External
external:
metricName: kafka_lag_metric_sm0ke
metricSelector:
matchLabels:
topic: prices
targetValue: 5
HPA 信息
kubectl describe hpa kafkademo-hpa
Name: kafkademo-hpa
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Sat, 17 Apr 2021 20:01:29 +0300
Reference: Deployment/kafkademo
Metrics: ( current / target )
"kafka_lag_metric_sm0ke" (target value): <unknown> / 5
Min replicas: 3
Max replicas: 12
Deployment pods: 3 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 70m (x335 over 155m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get external metric kafka_lag_metric_sm0ke: unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
Warning FailedGetExternalMetric 2m30s (x366 over 155m) horizontal-pod-autoscaler unable to get external metric default/kafka_lag_metric_sm0ke/&LabelSelector{MatchLabels:map[string]string{topic: prices,},MatchExpressions:[]LabelSelectorRequirement{},}: no metrics returned from external metrics API
-- 编辑1
当我查询默认命名空间时，我得到这个：
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/kafka_lag_metric_sm0ke |jq
{
"kind": "ExternalMetricValueList",
"apiVersion": "external.metrics.k8s.io/v1beta1",
"metadata": {},
"items": []
}
我可以看到“项目”字段是空的。这是什么意思？
我似乎不理解幕后发生的一系列事件。
据我所知，这就是发生的事情。 这是正确的吗？
- prometheus-adapter 查询 Prometheus，执行 seriesQuery，计算 metricsQuery 并创建“kafka_lag_metric_sm0ke”
- 它向 api 服务器注册一个端点以获取外部指标。
- API 服务器将根据该端点定期更新其统计信息。
- HPA 从 API 服务器检查“kafka_lag_metric_sm0ke”并根据提供的值执行缩放。
我似乎也不明白名称空间在这一切中的重要性。我可以看到该统计信息已命名空间。这是否意味着每个命名空间将有 1 个统计信息？这有什么意义？
在我提出问题后回答我自己的问题的悠久传统，以上配置有什么问题。
错误在prometheus-adapter yaml:
rules:
external:
- seriesQuery: 'kafka_consumergroup_group_lag{topic="prices"}'
resources:
template: <<.Resource>>
name:
as: "kafka_lag_metric_sm0ke"
metricsQuery: 'avg by (topic) (round(avg_over_time(<<.Series>>{<<.LabelMatchers>>}[1m])))'
我删除了 <<.LabelMatchers>>，现在可以使用了：
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/kafka_lag_metric_sm0ke |jq
{
"kind": "ExternalMetricValueList",
"apiVersion": "external.metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metricName": "kafka_lag_metric_sm0ke",
"metricLabels": {
"topic": "prices"
},
"timestamp": "2021-04-21T16:55:18Z",
"value": "0"
}
]
}
我仍然不确定它为什么有效。我知道 <<.LabelMatchers>> 在这种情况下将被替换为不会产生有效查询的内容，但我不知道它是什么。
@sm0ke21
不支持 hpa api 版本 autoscaling/v2beta1，您应该使用 autoscaling/v2beta2 或新的 api 版本