尝试从 Apache Beam 访问 Google Cloud Storage 时出现 HttpForbiddenError

Question

我正在尝试使用 Apache Beam 从 Compute Engine VM 简单访问 Google 云存储。当然，我已经使用命令设置了默认的应用程序登录 gcloud auth 应用程序-默认登录并为计算引擎服务帐户添加对存储的访问。我已经运行使用 DirectRunner 的管道并得到错误： apache_beam.io.filesystem.BeamIOError：匹配操作失败，出现异常 {'gs://{THIS MY BUCKETNAME}/source/sales_transactions.csv'：HttpForbiddenError()}

#import print library
import logging

#import apache beam library
import apache_beam as beam

#import pipeline options.
from apache_beam.options.pipeline_options import  PipelineOptions

#Create a pipeline
plOps = beam.Pipeline(options=PipelineOptions())

#--------------------------------------------------
# 1.Read from a text file.
#--------------------------------------------------

#Read the file from Google Cloud Storage
transactions = ( plOps 
                | 'Read Transaction CSV'
                    >> beam.io.ReadFromText('gs://{THIS MY BUCKETNAME}/data/sales_transactions.csv')
                )

printSize(transactions,'Raw Transactions')

Answer 1

我已经部分解决了这个问题 - 阅读正常。我已经在 DEVELOPER MACHINE 上使用 sudo su 以 root 身份登录，Apache Beam 获得了从 GS 读取文件的权限。但是当我尝试像这样写入 GS 存储桶时

#Write output to a text file
( custTypeCount | 'Write to GS Text'
        >> beam.io.WriteToText('gs://{MY BUCKET NAME}/output/customertype-summary.txt')
)

脚本出错：

RuntimeError: HttpForbiddenError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/{MY BUCKET NAME}/o?uploadType=resumable&alt=json&name=output%2Fbeam-temp-customertype-summary.txt-7bea505ad0bf11e9b69c42010a800002%2F55a9057e-18e5-4171-9db4-9e55601b2a8d.customertype-summary.txt>: response: <{'status': '403', 'content-length': '208', 'vary': 'Origin, X-Origin', 'server': 'UploadServer', 'x-guploader-uploadid': 'AEnB2Upo4RBzVV1S51_uWhcCiK_uK_iOSRAdAb8HWMhxznaPr0JcHKWxKDLwHbtTIYvHuMjyESV4dZqAfN3TaWYMqr5gQeypcQ', 'date': 'Fri, 06 Sep 2019 16:00:44 GMT', 'content-type': 'application/json; charset=UTF-8', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"'}>, content <{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "insufficientPermissions",
    "message": "Insufficient Permission"
   }
  ],
  "code": 403,
  "message": "Insufficient Permission"
 }
}
> [while running 'Write to GS Text/Write/WriteImpl/WriteBundles']

而且我不知道应该添加什么权限

Answer 2

是啊！我解决了。只需重新创建默认应用程序凭据——我真的不知道它们出了什么问题。我已经毫不含糊地做到了。否则文件夹 /root/.config/gcloud 不存在。但是凭据发生了一些事情。

那么，如何解决呢： 1.须藤须 2.拖放文件夹 /root/.config/gcloud 3. 运行再次gcloud auth application-default login

走运！

尝试从 Apache Beam 访问 Google Cloud Storage 时出现 HttpForbiddenError

HttpForbiddenError when trying to access Google Cloud Storage from Apache Beam

google-cloud-storage

google-cloud-platform

google-cloud-dataflow

apache-beam-io