使用 'monitoring_v3' 为 GKE 中的容器创建自定义指标时出错

Question

我们收到以下错误

"One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0]"

当我们尝试在 GKE 中创建容器级别指标时。我们正在创建自定义指标以每秒更新 app_queue 长度。它在正常情况下运行良好。但是，当 pods 超过 100 时 - 我们会收到一些 API 调用的错误，而且错误的数量非常多。

问题： 我们缺少任何配额或设置吗？或者一些创建自定义指标的最佳实践？

堆栈跟踪：

      File "/usr/local/lib/python3.6/dist-packages/Base-1.0-py3.6.egg/base/custom_metrics.py", line 63, in create_custom_metric
        client.create_time_series(project_name, [series])
      File "/usr/local/lib/python3.6/dist-packages/google/cloud/monitoring_v3/gapic/metric_service_client.py", line 775, in create_time_series
        request, retry=retry, timeout=timeout, metadata=metadata)
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
        return wrapped_func(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
        on_error=on_error,
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 177, in retry_target
        return target()
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/timeout.py", line 206, in func_with_timeout
        return func(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/grpc_helpers.py", line 61, in error_remapped_callable
        six.raise_from(exceptions.from_grpc_error(exc), exc)
      File "<string>", line 3, in raise_from
    google.api_core.exceptions.InvalidArgument: 400 One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0]

Answer 1

如果您达到任何配额限制，Stackdriver Logging 中应记录错误。您还可以访问您的配额页面，查看是否达到任何限制 [菜单>IAM 和管理>配额]。

关于要遵循的规则，请查看此 documentation 以更深入地了解如何将数据写入自定义指标类型。将多个数据点添加到同一时间序列时的最佳做法是添加速度不要超过每分钟一次。

使用 'monitoring_v3' 为 GKE 中的容器创建自定义指标时出错

Error while creating custom metrics with 'monitoring_v3' for containers in GKE

google-cloud-platform

google-kubernetes-engine

google-cloud-monitoring