在 Google Composer 的气流实例上安装 apache-airflow-backport-providers-google 模块时出现问题

Question

我需要从 Composer 执行数据融合管道，为此使用 de operatos:

from airflow.providers.google.cloud.operators.datafusion import (
    CloudDataFusionCreateInstanceOperator,
    CloudDataFusionCreatePipelineOperator,
    CloudDataFusionDeleteInstanceOperator,
    CloudDataFusionDeletePipelineOperator,
    CloudDataFusionGetInstanceOperator,
    CloudDataFusionListPipelinesOperator,
    CloudDataFusionRestartInstanceOperator,
    CloudDataFusionStartPipelineOperator,
    CloudDataFusionStopPipelineOperator,
    CloudDataFusionUpdateInstanceOperator,
)

我遇到的问题是关于模“apache-airflow-backport-providers-google”，在这个链接的支持下我知道我需要使用这个模：

在气流实例中安装模数的参考（由@Gonzalo Pérez Fernández 回答）：https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/datafusion.html

当我尝试安装 python 对 Composer 的依赖项（如 PyPi 包）时，出现此错误：

UPDATE operation on this environment failed 7 minutes ago with the following error message:
Failed to install PyPI packages. 
apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0. Check the Cloud Build log at https://console.cloud.google.com/cloud-build/builds/a2ecf37a-4c47-4770-9489-6fb65e87d82f?project=341768372632 for details. For detailed instructions see https://cloud.google.com/composer/docs/troubleshooting-package-installation

日志细节是：

apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0.
apache-airflow-backport-providers-google 2021.3.3 has requirement apache-airflow~=1.10, but you have apache-airflow 2.1.2+composer.
The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION  fail' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

有没有什么方法可以使用 de module "apache-airflow-backport-providers-google" 而不会在 composer 实例上出现依赖性问题？或者使用数据融合运算符的最佳方法是什么不需要更改或解析包版本 python?.

使用的 Composer 映像版本： composer-1.17.0-airflow-2.1.2

谢谢。

Answer 1

Airflow 2.0+ 无需安装 apache-airflow-backport-providers-google。该软件包实际上将 Airflow 2 运算符反向移植到 Airflow 1.10.*。此外，在 Composer 版本 composer-1.17.0-airflow-2.1.2 中，apache-airflow-providers-google==5.0.0 包已根据 documentation 安装。您应该能够使用您按原样发布的代码片段导入 Data Fusion 运算符。

但是，如果不是这种情况，您可能应该在尝试重新安装时处理日志中显示的冲突 apache-airflow-providers-google==5.0.0:

apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0.

您可以在 PyPi 依赖项中添加 google-ads=12.0.0 的要求，看看它是否有效。

在 Google Composer 的气流实例上安装 apache-airflow-backport-providers-google 模块时出现问题

Issue installing apache-airflow-backport-providers-google module on airflow instance of Google Composer

airflow

google-cloud-composer

google-cloud-data-fusion