GCP:无法通过 gcloud/API 创建项目接收器
GCP: Unable to create Project Sink through gcloud/API
我目前有一个场景,其中我使用 Deployment Manager 创建了部署脚本 (python),它 运行 在我倾向于尝试使用的 GCP 沙箱 (LA Playground) 中非常好事情出来了。但是,当我尝试 运行 在具有计费帐户等的实际 GCP 项目上使用它时,即使我在为它创建的服务帐户中使用基本相同的角色集,我也会收到权限错误。我也是那个麻烦的 GCP 帐户的项目所有者。
我遇到问题的特定权限集是创建项目接收器。我总是收到以下错误:
Error in Operation [operation-xxxxxxxxxxxx-xxxxxxxxxxxxx-xxxxxxxx-9d487530]: errors:
- code: RESOURCE_ERROR
location: /deployments/structured-pipeline/resources/dataprep-bq-listener-sink
message: '{"ResourceType":"gcp-types/logging-v2:projects.sinks","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"message":"The
caller does not have permission","status":"PERMISSION_DENIED","statusMessage":"Forbidden","requestPath":"https://logging.googleapis.com/v2/projects/redacted-gcp-project-name-dev/sinks","httpMethod":"POST"}}'
这是我在用于 运行 部署的服务帐户上附加的角色列表:
- 计算管理员
- 部署管理器编辑器
- 安全管理员
- 日志管理员
- 日志配置编写器
- Pub/Sub 管理员
- 服务使用管理员
- 存储管理员
如您所见,我已经将他们设置为管理员级别,以确保我没有缺少所需的权限。
根据官方文档,创建项目接收器所需的特定权限是 logging.sinks.create
,它包含在 Logging Admin
和 Logs Configuration Writer
角色中。
更新#1:
这是问题资源的扩展DM配置
- name: dataprep-bq-listener-sink
properties:
destination: pubsub.googleapis.com/projects/redacted-gcp-project-name-dev/topics/dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh
filter: |-
resource.type="bigquery_dataset"
resource.labels.dataset_id="dataprep_output_vpcps3s6wjzbmkbxxlsqqh"
protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"
outputVersionFormat: V2
sink: dataprep-bq-listener-sink-vpcps3s6wjzbmkbxxlsqqh
type: gcp-types/logging-v2:projects.sinks
- accessControl:
gcpIamPolicy:
bindings:
- members:
- $(ref.dataprep-bq-listener-sink.writerIdentity)
role: roles/pubsub.publisher
name: dataprep-bq-listener
properties:
labels:
client_namespace: redacted-client-name
topic: dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh
type: pubsub.v1.topic
metadata:
dependsOn:
- dataprep-bq-listener-sink
更新#2
我将 gcloud
设置为使用服务帐户作为凭据 (https://cloud.google.com/sdk/docs/authorizing#authorizing_with_a_service_account),该帐户具有我上面所述的角色。
运行 gcloud auth list
明明指向服务账号我 created/used.
ACTIVE ACCOUNT
my_redacted_email@company.com
cloud_user_p_2725ef46@linuxacademygclabs.com
cloud_user_p_2e3db20d@linuxacademygclabs.com
cloud_user_p_41b5121a@linuxacademygclabs.com
cloud_user_p_b57aaef8@linuxacademygclabs.com
cloud_user_p_bdb72060@linuxacademygclabs.com
cloud_user_p_c2f5d19a@linuxacademygclabs.com
cloud_user_p_c3c54122@linuxacademygclabs.com
cloud_user_p_c88350f1@linuxacademygclabs.com
cloud_user_p_d7702b8b@linuxacademygclabs.com
* service_account_used_in_dm@project_id.iam.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
更新#3
运行 gcloud projects get-iam-policy <PROJECT_ID>
结果到以下 IAM 策略列表 (我已将我的电子邮件替换为“my_redacted_email@company.com”和我用于 DM 的服务帐户电子邮件“service_account_used_in_dm@project_id.iam.gserviceaccount.com”;所有默认的默认服务帐户我保留了)
bindings:
- members:
- user: my_redacted_email@company.com
role: roles/billing.projectManager
- members:
- serviceAccount:1008104628570@cloudbuild.gserviceaccount.com
role: roles/cloudbuild.builds.builder
- members:
- serviceAccount:service-1008104628570@gcp-sa-cloudbuild.iam.gserviceaccount.com
role: roles/cloudbuild.serviceAgent
- members:
- serviceAccount:service-1008104628570@gcf-admin-robot.iam.gserviceaccount.com
role: roles/cloudfunctions.serviceAgent
- members:
- serviceAccount:service-1008104628570@gcp-sa-cloudscheduler.iam.gserviceaccount.com
role: roles/cloudscheduler.serviceAgent
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/compute.admin
- members:
- serviceAccount:service-1008104628570@compute-system.iam.gserviceaccount.com
role: roles/compute.serviceAgent
- members:
- serviceAccount:service-1008104628570@containerregistry.iam.gserviceaccount.com
role: roles/containerregistry.ServiceAgent
- members:
- serviceAccount:service-1008104628570@dataflow-service-producer-prod.iam.gserviceaccount.com
role: roles/dataflow.serviceAgent
- members:
- serviceAccount:service-1008104628570@trifacta-gcloud-prod.iam.gserviceaccount.com
role: roles/dataprep.serviceAgent
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/deploymentmanager.editor
- members:
- serviceAccount:1008104628570-compute@developer.gserviceaccount.com
- serviceAccount:1008104628570@cloudservices.gserviceaccount.com
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
- user:my_redacted_email@company.com
role: roles/editor
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/iam.securityAdmin
- members:
- user:my_redacted_email@company.com
role: roles/iam.serviceAccountUser
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/logging.admin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/logging.configWriter
- members:
- user:my_redacted_email@company.com
role: roles/owner
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/pubsub.admin
- members:
- user:my_redacted_email@company.com
role: roles/resourcemanager.projectIamAdmin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/serviceusage.serviceUsageAdmin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/storage.admin
etag: BwW0MM3sGXk=
version: 1
您可能已经注意到,我使用的服务帐户绑定了我上面在项目级别提到的角色。
更新#4
使用相同的服务帐户通过 gcloud
创建接收器会成功创建接收器。
gcloud logging sinks create dataprep-bq-listener-sink-vpcps3s6wjzbmkbxxlsqqh pubsub.googleapis.com/projects/<project_id>/topics/dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh --log-filter='resource.type="bigquery_dataset" AND resource.labels.dataset_id="dataprep_output_vpcps3s6wjzbmkbxxlsqqh" AND protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"' --project=<project_id>
更新#5
@Kolban 注意到 Deployment Manager 对 [PROJECT_NUMBER]@cloudservices.gserviceaccount.com
服务帐户对 Editor
角色 per stated in their official documentation 的要求,现在查看 UPDATE#4 ,它清楚地表明该服务帐户具有 Editor
角色。
他还指出可能会混淆多个云帐户(我可能 运行 使用服务帐户但在不同的云帐户下执行命令),因此,我执行了以下命令:gcloud config get-value account
& gcloud config get-value project
结果是我所期望的,这是正确的 GCP 项目和服务帐户对。
docs状态:-
To create other Google Cloud resources, Deployment Manager uses the
credentials of the Google APIs service account to authenticate to
other APIs. The Google APIs service account is designed specifically
to run internal Google processes on your behalf. The service account
is identifiable using the email:
[PROJECT_NUMBER]@cloudservices.gserviceaccount.com
在您的情况下,您的 1008104628570@cloudservices.gserviceaccount.com
服务帐户在项目级别绑定到 roles\editor
角色。但是,您需要的权限 (logging.sinks.create
) 不包含在该旧编辑者角色中。
您能否尝试另外授予 1008104628570@cloudservices.gserviceaccount.com
服务帐户 Logging Admin 角色 (roles/logging.admin
),看看是否有帮助?
我目前有一个场景,其中我使用 Deployment Manager 创建了部署脚本 (python),它 运行 在我倾向于尝试使用的 GCP 沙箱 (LA Playground) 中非常好事情出来了。但是,当我尝试 运行 在具有计费帐户等的实际 GCP 项目上使用它时,即使我在为它创建的服务帐户中使用基本相同的角色集,我也会收到权限错误。我也是那个麻烦的 GCP 帐户的项目所有者。
我遇到问题的特定权限集是创建项目接收器。我总是收到以下错误:
Error in Operation [operation-xxxxxxxxxxxx-xxxxxxxxxxxxx-xxxxxxxx-9d487530]: errors:
- code: RESOURCE_ERROR
location: /deployments/structured-pipeline/resources/dataprep-bq-listener-sink
message: '{"ResourceType":"gcp-types/logging-v2:projects.sinks","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"message":"The
caller does not have permission","status":"PERMISSION_DENIED","statusMessage":"Forbidden","requestPath":"https://logging.googleapis.com/v2/projects/redacted-gcp-project-name-dev/sinks","httpMethod":"POST"}}'
这是我在用于 运行 部署的服务帐户上附加的角色列表:
- 计算管理员
- 部署管理器编辑器
- 安全管理员
- 日志管理员
- 日志配置编写器
- Pub/Sub 管理员
- 服务使用管理员
- 存储管理员
如您所见,我已经将他们设置为管理员级别,以确保我没有缺少所需的权限。
根据官方文档,创建项目接收器所需的特定权限是 logging.sinks.create
,它包含在 Logging Admin
和 Logs Configuration Writer
角色中。
更新#1:
这是问题资源的扩展DM配置
- name: dataprep-bq-listener-sink
properties:
destination: pubsub.googleapis.com/projects/redacted-gcp-project-name-dev/topics/dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh
filter: |-
resource.type="bigquery_dataset"
resource.labels.dataset_id="dataprep_output_vpcps3s6wjzbmkbxxlsqqh"
protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"
outputVersionFormat: V2
sink: dataprep-bq-listener-sink-vpcps3s6wjzbmkbxxlsqqh
type: gcp-types/logging-v2:projects.sinks
- accessControl:
gcpIamPolicy:
bindings:
- members:
- $(ref.dataprep-bq-listener-sink.writerIdentity)
role: roles/pubsub.publisher
name: dataprep-bq-listener
properties:
labels:
client_namespace: redacted-client-name
topic: dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh
type: pubsub.v1.topic
metadata:
dependsOn:
- dataprep-bq-listener-sink
更新#2
我将 gcloud
设置为使用服务帐户作为凭据 (https://cloud.google.com/sdk/docs/authorizing#authorizing_with_a_service_account),该帐户具有我上面所述的角色。
运行 gcloud auth list
明明指向服务账号我 created/used.
ACTIVE ACCOUNT
my_redacted_email@company.com
cloud_user_p_2725ef46@linuxacademygclabs.com
cloud_user_p_2e3db20d@linuxacademygclabs.com
cloud_user_p_41b5121a@linuxacademygclabs.com
cloud_user_p_b57aaef8@linuxacademygclabs.com
cloud_user_p_bdb72060@linuxacademygclabs.com
cloud_user_p_c2f5d19a@linuxacademygclabs.com
cloud_user_p_c3c54122@linuxacademygclabs.com
cloud_user_p_c88350f1@linuxacademygclabs.com
cloud_user_p_d7702b8b@linuxacademygclabs.com
* service_account_used_in_dm@project_id.iam.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
更新#3
运行 gcloud projects get-iam-policy <PROJECT_ID>
结果到以下 IAM 策略列表 (我已将我的电子邮件替换为“my_redacted_email@company.com”和我用于 DM 的服务帐户电子邮件“service_account_used_in_dm@project_id.iam.gserviceaccount.com”;所有默认的默认服务帐户我保留了)
bindings:
- members:
- user: my_redacted_email@company.com
role: roles/billing.projectManager
- members:
- serviceAccount:1008104628570@cloudbuild.gserviceaccount.com
role: roles/cloudbuild.builds.builder
- members:
- serviceAccount:service-1008104628570@gcp-sa-cloudbuild.iam.gserviceaccount.com
role: roles/cloudbuild.serviceAgent
- members:
- serviceAccount:service-1008104628570@gcf-admin-robot.iam.gserviceaccount.com
role: roles/cloudfunctions.serviceAgent
- members:
- serviceAccount:service-1008104628570@gcp-sa-cloudscheduler.iam.gserviceaccount.com
role: roles/cloudscheduler.serviceAgent
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/compute.admin
- members:
- serviceAccount:service-1008104628570@compute-system.iam.gserviceaccount.com
role: roles/compute.serviceAgent
- members:
- serviceAccount:service-1008104628570@containerregistry.iam.gserviceaccount.com
role: roles/containerregistry.ServiceAgent
- members:
- serviceAccount:service-1008104628570@dataflow-service-producer-prod.iam.gserviceaccount.com
role: roles/dataflow.serviceAgent
- members:
- serviceAccount:service-1008104628570@trifacta-gcloud-prod.iam.gserviceaccount.com
role: roles/dataprep.serviceAgent
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/deploymentmanager.editor
- members:
- serviceAccount:1008104628570-compute@developer.gserviceaccount.com
- serviceAccount:1008104628570@cloudservices.gserviceaccount.com
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
- user:my_redacted_email@company.com
role: roles/editor
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/iam.securityAdmin
- members:
- user:my_redacted_email@company.com
role: roles/iam.serviceAccountUser
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/logging.admin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/logging.configWriter
- members:
- user:my_redacted_email@company.com
role: roles/owner
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/pubsub.admin
- members:
- user:my_redacted_email@company.com
role: roles/resourcemanager.projectIamAdmin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/serviceusage.serviceUsageAdmin
- members:
- serviceAccount:service_account_used_in_dm@project_id.iam.gserviceaccount.com
role: roles/storage.admin
etag: BwW0MM3sGXk=
version: 1
您可能已经注意到,我使用的服务帐户绑定了我上面在项目级别提到的角色。
更新#4
使用相同的服务帐户通过 gcloud
创建接收器会成功创建接收器。
gcloud logging sinks create dataprep-bq-listener-sink-vpcps3s6wjzbmkbxxlsqqh pubsub.googleapis.com/projects/<project_id>/topics/dataprep-bq-listener-vpcps3s6wjzbmkbxxlsqqh --log-filter='resource.type="bigquery_dataset" AND resource.labels.dataset_id="dataprep_output_vpcps3s6wjzbmkbxxlsqqh" AND protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"' --project=<project_id>
更新#5
@Kolban 注意到 Deployment Manager 对 [PROJECT_NUMBER]@cloudservices.gserviceaccount.com
服务帐户对 Editor
角色 per stated in their official documentation 的要求,现在查看 UPDATE#4 ,它清楚地表明该服务帐户具有 Editor
角色。
他还指出可能会混淆多个云帐户(我可能 运行 使用服务帐户但在不同的云帐户下执行命令),因此,我执行了以下命令:gcloud config get-value account
& gcloud config get-value project
结果是我所期望的,这是正确的 GCP 项目和服务帐户对。
docs状态:-
To create other Google Cloud resources, Deployment Manager uses the credentials of the Google APIs service account to authenticate to other APIs. The Google APIs service account is designed specifically to run internal Google processes on your behalf. The service account is identifiable using the email:
[PROJECT_NUMBER]@cloudservices.gserviceaccount.com
在您的情况下,您的 1008104628570@cloudservices.gserviceaccount.com
服务帐户在项目级别绑定到 roles\editor
角色。但是,您需要的权限 (logging.sinks.create
) 不包含在该旧编辑者角色中。
您能否尝试另外授予 1008104628570@cloudservices.gserviceaccount.com
服务帐户 Logging Admin 角色 (roles/logging.admin
),看看是否有帮助?