GCP AI 平台 API - Class 级别的对象检测指标 (Python)

Question

我在 Vertex AI（GCP AI Platform 下的一项服务）中训练了一个 AutoML 对象检测模型。我正在尝试针对不同的置信度分数阈值和 IoU 阈值访问每个标签的模型评估指标（精度、召回率、准确性等）。

但是，我停留在第一步，即使是为了获得模型的综合性能指标，也比粒度级别的性能指标要少得多。我已经关注了this instruction But I cannot seem to figure out what is evaluation_id (also see the official sample code snippet here)，也就是：

def get_model_evaluation_image_object_detection_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

一段时间后，我发现对于在欧盟训练的模型，api_endpoint 应传递为：

location: str = "europe-west4"
api_endpoint: str = "europe-west4-aiplatform.googleapis.com"

但是无论我尝试什么 evaluation_id 都会导致以下错误：

InvalidArgument: 400 List of found errors:  1.Field: name; Message: Invalid ModelEvaluation resource name.

在文档中它说（这似乎包含我需要的东西）：

For the bounding box metric, Vertex AI returns an array of metric values at different IoU threshold values (between 0 and 1) and confidence threshold values (between 0 and 1). For example, you can narrow in on evaluation metrics at an IoU threshold of 0.85 and a confidence threshold of 0.8228. By viewing these different threshold values, you can see how they affect other metrics such as precision and recall.

在不知道输出数组中包含的情况下，如何对每个 class 起作用？基本上我需要每个 class 不同 IoU 阈值和置信度阈值的模型指标。

我也尝试从 AutoML API 查询，例如：

client_options = {'api_endpoint': 'eu-automl.googleapis.com:443'}

client = automl.AutoMlClient(client_options=client_options)
# Get the full path of the model.
model_full_id = client.model_path(project_id, "europe-west4", model_id)

print("List of model evaluations:")
for evaluation in client.list_model_evaluations(parent=model_full_id, filter=""):
    print("Model evaluation name: {}".format(evaluation.name))
    print("Model annotation spec id: {}".format(evaluation.annotation_spec_id))
    print("Create Time: {}".format(evaluation.create_time))
    print("Evaluation example count: {}".format(evaluation.evaluated_example_count))
    print(
        "Classification model evaluation metrics: {}".format(
            evaluation.classification_evaluation_metrics
        )
    )

毫不奇怪，这也行不通，并导致：

InvalidArgument: 400 List of found errors:  1.Field: parent; Message: The provided location ID doesn't match the endpoint. For automl.googleapis.com, the valid location ID is `us-central1`. For eu-automl.googleapis.com, the valid location ID is `eu`.

Answer 1

我能够使用 aiplatform_v1 which is well documented and this is the reference linked from the Vertex AI reference page 获得模型评估的响应。

在这个脚本中，我运行 list_model_evaluations() to get the evaluation name and used it as input for get_model_evaluation() 将 return 置信度分数阈值、IoU 阈值等的评估详细信息

注意：我在 europe-west4 中没有经过训练的模型，所以我改用了 us-central1。但是如果你接受过 europe-west4 的培训，你应该按照 location document.

使用 https://europe-west4-aiplatform.googleapis.com 作为 api_endpoint

from google.cloud import aiplatform_v1 as aiplatform

api_endpoint = 'us-central1-aiplatform.googleapis.com'
client_options = {"api_endpoint": api_endpoint}
client_model = aiplatform.services.model_service.ModelServiceClient(client_options=client_options)
project_id = 'your-project-id'
location = 'us-central1'
model_id = '999999999999'

model_name = f'projects/{project_id}/locations/{location}/models/{model_id}'
list_eval_request = aiplatform.types.ListModelEvaluationsRequest(parent=model_name)
list_eval = client_model.list_model_evaluations(request=list_eval_request)
eval_name=''
for val in list_eval:
    eval_name = val.name

get_eval_request = aiplatform.types.GetModelEvaluationRequest(name=eval_name)
get_eval = client_model.get_model_evaluation(request=get_eval_request)
print(get_eval)

查看回复片段：

name: "projects/xxxxxxxxx/locations/us-central1/models/999999999999/evaluations/1234567890"
metrics_schema_uri: "gs://google-cloud-aiplatform/schema/modelevaluation/image_object_detection_metrics_1.0.0.yaml"
metrics {
  struct_value {
    fields {
      key: "boundingBoxMeanAveragePrecision"
      value {
        number_value: 0.20201288
      }
    }
    fields {
      key: "boundingBoxMetrics"
      value {
        list_value {
          values {
            struct_value {
              fields {
                key: "confidenceMetrics"
                value {
                  list_value {
                    values {
                      struct_value {
                        fields {
                          key: "confidenceThreshold"
                          value {
                            number_value: 0.06579724
                          }
                        }
                        fields {
                          key: "f1Score"
                          value {
                            number_value: 0.15670435
                          }
                        }
                        fields {
                          key: "precision"
                          value {
                            number_value: 0.09326923
                          }
                        }
                        fields {
                          key: "recall"
                          value {
                            number_value: 0.48989898
                          }
                        }
                      }
                    }
                    values {
                      struct_value {
....

编辑 1：根据 class

获取响应

要获取每个 class 的指标，您可以使用 list_model_evaluation_slices() to get the name for each class, then use the name to get_model_evaluation_slice()。在此代码中，我将名称推送到列表中，因为我有多个 classes。然后只需使用存储在数组中的值来获取每个 class.

的指标

在我的代码中，我使用 label[0] 从这个 class 获得一个单一的响应。

from google.cloud import aiplatform_v1 as aiplatform

api_endpoint = 'us-central1-aiplatform.googleapis.com'
client_options = {"api_endpoint": api_endpoint}
client_model = aiplatform.services.model_service.ModelServiceClient(client_options=client_options)
project_id = 'your-project-id'
location = 'us-central1'
model_id = '999999999999'

model_name = f'projects/{project_id}/locations/{location}/models/{model_id}'
list_eval_request = aiplatform.types.ListModelEvaluationsRequest(parent=model_name)
list_eval = client_model.list_model_evaluations(request=list_eval_request)
eval_name=''
for val in list_eval:
    eval_name = val.name

label=[]
slice_eval_request = aiplatform.types.ListModelEvaluationSlicesRequest(parent=eval_name)
slice_eval = client_model.list_model_evaluation_slices(request=slice_eval_request)
for data in slice_eval:
    label.append(data.name)

get_eval_slice_request = aiplatform.types.GetModelEvaluationSliceRequest(name=label[0])
get_eval_slice = client_model.get_model_evaluation_slice(request=get_eval_slice_request)
print(get_eval_slice)

打印所有 classes：

类在 UI 中：

class 的响应片段：

name: "projects/xxxxxxxxx/locations/us-central1/models/999999999/evaluations/0000000000/slices/777777777"
slice_ {
  dimension: "annotationSpec"
  value: "Cheese"
}
metrics_schema_uri: "gs://google-cloud-aiplatform/schema/modelevaluation/image_object_detection_metrics_1.0.0.yaml"
metrics {
  struct_value {
    fields {
      key: "boundingBoxMeanAveragePrecision"
      value {
        number_value: 0.14256561
      }
    }
    fields {
      key: "boundingBoxMetrics"
      value {
        list_value {
          values {
            struct_value {
              fields {
                key: "confidenceMetrics"
                value {
                  list_value {
                    values {
                      struct_value {
                        fields {
                          key: "confidenceThreshold"
                          value {
                            number_value: 0.06579724
                          }
                        }
                        fields {
                          key: "f1Score"
                          value {
                            number_value: 0.10344828
                          }
                        }
                        fields {
                          key: "precision"
                          value {
                            number_value: 0.06198347
                          }
                        }
....

GCP AI 平台 API - Class 级别的对象检测指标 (Python)

GCP AI Platform API - Object Detection Metrics at Class Level (Python)

google-cloud-platform

google-cloud-automl

google-cloud-vertex-ai