Airflow BigQueryOperator DAG doesn't let me query a bigquery-public-dataset: Error: accessDenied

Question

我已经在 Google Cloud Composer 中设置了一个环境，并为关联的服务帐户授予了 BigQuery 上的管理员角色。

我的 DAG 看起来像这样：

from airflow.contrib.operators.bigquery_operator import BigQueryOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'My name',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}
gs_bucket = 'europe-central2-my-name-de-4237c541-bucket'

dag = DAG('Test-DAG-Two',
          start_date=datetime(2022, 1, 16),
          schedule_interval='0 8 * * *',
          concurrency=5,
          max_active_runs=1,
          default_args=default_args)

create_test_table_2 = BigQueryOperator(
  task_id='t_create_test_table_2',
  sql=""" SELECT * FROM `bigquery-public-data.crypto_ethereum.blocks` WHERE DATE(timestamp) = "2022-01-19" LIMIT 1000 """,
  destination_dataset_table='myproject.test_data_set.test_table_2${{ ds_nodash }}',
  write_disposition='WRITE_TRUNCATE',
  bigquery_conn_id='bigquery_default',
  google_cloud_storage_conn_id='bigquery_default',
  use_legacy_sql=False,
  dag=dag
)

当我运行相同的 DAG 但使 SQL 查询查询我自己的数据集之一时，它按预期工作。但是，当我运行这个 DAG 我得到这个错误：

错误 - BigQuery 作业失败。最终错误是：{'reason': 'accessDenied', 'message': 'Access Denied: Table bigquery-public-data:crypto_ethereum.blocks: User does not have permission to query table bigquery-public-data:crypto_ethereum.blocks.'}

Answer 1

现在已经解决了。问题是 destination_dataset_table 位于欧盟，但 bigquery-public-data.crypto_ethereum.blocks 位于美国。我必须创建一个位于美国的新目标数据集才能正常工作。

Airflow BigQueryOperator DAG doesn't let me query a bigquery-public-dataset: Error: accessDenied

Airflow BigQueryOperator DAG doesn't let me query a bigquery-public-dataset: Error: accessDenied

cloud

google-bigquery

airflow