在 Vertex AI 的日程表笔记本中授予用户 bigquery.datasets.create 的权限

Question

我有一个笔记本，我在其中通过 API 访问数据、处理数据并将结果发送到 BigQuery table 以及 GCS 存储桶。当手动运行笔记本时，一切正常。

但是，在计划时，它中断并显示以下消息：

/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in to_gbq(self, destination_table, project_id, chunksize, reauth, if_exists, auth_local_webserver, table_schema, location, progress_bar, credentials)
   1938             location=location,
   1939             progress_bar=progress_bar,
-> 1940             credentials=credentials,
   1941         )
   1942 

/opt/conda/lib/python3.7/site-packages/pandas/io/gbq.py in to_gbq(dataframe, destination_table, project_id, chunksize, reauth, if_exists, auth_local_webserver, table_schema, location, progress_bar, credentials)
    221         location=location,
    222         progress_bar=progress_bar,
--> 223         credentials=credentials,
    224     )

/opt/conda/lib/python3.7/site-packages/pandas_gbq/gbq.py in to_gbq(dataframe, destination_table, project_id, chunksize, reauth, if_exists, auth_local_webserver, table_schema, location, progress_bar, credentials, api_method, verbose, private_key)
   1155             credentials=connector.credentials,
   1156         )
-> 1157         table_connector.create(table_id, table_schema)
   1158     else:
   1159         original_schema = pandas_gbq.schema.to_pandas_gbq(table.schema)

/opt/conda/lib/python3.7/site-packages/pandas_gbq/gbq.py in create(self, table_id, schema)
   1311             _Dataset(
   1312                 self.project_id, credentials=self.credentials, location=self.location,
-> 1313             ).create(self.dataset_id)
   1314 
   1315         table_ref = TableReference(

/opt/conda/lib/python3.7/site-packages/pandas_gbq/gbq.py in create(self, dataset_id)
   1413             self.client.create_dataset(dataset)
   1414         except self.http_error as ex:
-> 1415             self.process_http_error(ex)

/opt/conda/lib/python3.7/site-packages/pandas_gbq/gbq.py in process_http_error(ex)
    384             raise QueryTimeout("Reason: {0}".format(ex))
    385 
--> 386         raise GenericGBQException("Reason: {0}".format(ex))
    387 
    388     def download_table(

GenericGBQException: Reason: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/****************/datasets?prettyPrint=false: Access Denied: Project ******************: User does not have bigquery.datasets.create permission in project *****************.

（星星是项目 ID（顺便说一句：这不是我选择的项目 ID）。）

对于脚本的以下部分：

Table_grouped_to_bg.to_gbq('retailer_accuracy.testbin', Context.default().project_id,chunksize=10000,if_exists='append') #
                    Table_grouped_to_bg.to_csv('gs://retailer-sectionalized-labels-csv/'+'Table_grouped_' +str(Store_name)+'_'+str(Date)+'.csv',sep=';' ,encoding='utf-8-sig')
                    ax1 =Table_grouped.plot.bar(x="distance_bin", y="percentage", rot=70, title="Percentage in each bin,{}".format(Date), figsize=(15, 5));
                    x_offset = -0.01
                    y_offset = 0.02
                    for p in ax1.patches:
                        b = p.get_bbox()
                        val = "{:+.2f}".format(b.y1 + b.y0)        
                        ax1.annotate(val, ((b.x0 + b.x1)/2 + x_offset, b.y1 + y_offset))

                    fig = ax1.get_figure() 
                    
                    
                    def saving_figure(path_logdir):
                        fig = ax1.get_figure() 
                        fig_to_upload = plt.gcf()

                        # Save figure image to a bytes buffer
                        buf = io.BytesIO()
                        fig_to_upload.savefig(buf, format='png')
                        buf.seek(0)
                        image_as_a_string = base64.b64encode(buf.read())

                        # init GCS client and upload buffer contents
                        client = Storage.Client()
                        bucket = client.get_bucket('retailer-sectionalized-statistics-plot-png')
                        blob = bucket.blob('Accuracy_barplot_'+str(Store_name)+'_'+str(Date)+'.png')  # This defines the path where the file will be stored in the bucket
                        your_file_contents = blob.upload_from_string(image_as_a_string, content_type='image/png')
                    plt.show(block=True)
                    plt.show(block=True) 
....................saving_figure('Accuracy_barplot_'+str(Store_name)+'_'+str(Date)+'.png')

星星是项目 ID。

我明白，运行按计划完成项目不再是我。那么问题是，如何通过 GCP 实例将笔记本的权限传递给运行？

Answer 1

此问题与权限错误有关。根据您的要求，您可以在项目中创建一个服务帐户。服务帐户是特殊类型的帐户，用于调用 API 并用作应用程序的标识。您可以将您的用户 ID 分配给服务帐户并授予 BigQuery Data Editor and Notebook Admin role to the service account.For more information, you can check this documentation.

在 Vertex AI 的日程表笔记本中授予用户 bigquery.datasets.create 的权限

Grant user permission to bigquery.datasets.create in schedules notebook in Vertex AI

python

google-bigquery

google-cloud-vertex-ai