使用 gcloud CLI 在 Cloud Composer 上创建 Airflow 连接

Question

我正在尝试使用 gcloud CLI 在 Cloud Composer 上创建气流连接。我关注文档和运行以下评论。 https://cloud.google.com/composer/docs/how-to/managing/connections#creating_new_airflow_connections

gcloud composer environments run my-env \
  --project my-project \
  --location asia-northeast1 \
  connections -- --add \
  --conn_id=my-conn-id \
  --conn_type=google_cloud_platform \
  --conn_extra '{"extra\__google\_cloud\_platform\__project": "my-project", \
  "extra\__google\_cloud\_platform\__key_path": "/home/airflow/gcs/data/keys/my-key.json", \
  "extra\__google\_cloud\_platform\__scope": "https://www.googleapis.com/auth/cloud-platform"}'

kubeconfig entry generated for asia-northeast1-my-env-44718514-gke.
Executing within the following kubectl namespace: default
W0525 22:51:11.244104   93234 flags.go:39] conn_extra is DEPRECATED and will be removed in
a future version. Use conn-extra instead.
W0525 22:51:11.244246   93234 flags.go:39] conn_extra is DEPRECATED and will be removed in
a future version. Use conn-extra instead.
W0525 22:51:11.244256   93234 flags.go:39] conn_extra is DEPRECATED and will be removed in
a future version. Use conn-extra instead.

[2019-05-25 14:51:13,215] {settings.py:176} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=1800
[2019-05-25 14:51:13,598] {default_celery.py:80} WARNING - You have configured a
result_backend of redis://airflow-redis-service:6379/0, it is highly recommended to use an
alternative result_backend (i.e. a database).
[2019-05-25 14:51:13,600] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-05-25 14:51:13,680] {app.py:51} WARNING - Using default Composer Environment Variables.
Overrides have not been applied.
[2019-05-25 14:51:13,688] {configuration.py:516} INFO - Reading the config from
/etc/airflow/airflow.cfg
[2019-05-25 14:51:13,698] {configuration.py:516} INFO - Reading the config from 
/etc/airflow/airflow.cfg

连接已成功创建，但项目 ID、密钥文件路径和范围为空。所以连接无效。

当我手动创建时，那些属性不是空的。我错过了什么吗？

作曲家图像：composer-1.5.0-airflow-1.10.1

Answer 1

我无法复制它。当我运行以下命令时，连接确实添加了 extras 字段：

gcloud composer environments run my-env \
  --project my-project \
  --location europe-west1 \
  connections -- --add \
  --conn_id=my-conn-id \
  --conn_type=google_cloud_platform \
  --conn_extra='{"extra__google_cloud_platform__project": "my-project", "extra__google_cloud_platform__key_path":"/tesf"}

Answer 2

转义引号时发现一些语法错误。这很好用。

$ CONNECTION_CREATE_COMMAND="gcloud composer environments run $COMPOSER_ENVIRONMENT \
  --project $COMPOSER_PROJECT \
  --location ${COMPOSER_LOCATION} \
  connections -- --add \
  --conn_id=${CONN_ID_BASE}_${app}_${c} \
  --conn_type=google_cloud_platform \
  --conn_extra '{\"extra__google_cloud_platform__project\": \"${BQ_PROJECT}\", \
                 \"extra__google_cloud_platform__key_path\": \"${KEY_JSON_FILE_PATH}\", \
                 \"extra__google_cloud_platform__scope\": \"https://www.googleapis.com/auth/cloud-platform\"}'"
$ eval $CONNECTION_CREATE_COMMAND

使用 gcloud CLI 在 Cloud Composer 上创建 Airflow 连接

Create Airflow connections on Cloud Composer using gcloud CLI

gcloud

airflow

google-cloud-composer