Dataflow fails due to ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
Dataflow fails due to ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
自今年年初以来，我们的 python 数据流作业导致 worker 启动时出错：
Processing /var/opt/google/dataflow/setuptools-51.1.1.tar.gz
ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
ERROR: No matching distribution found for wheel
ERROR: Command errored out with exit status 1: /usr/local/bin/python3 /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-rvfx0hn9/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=40.8.0' wheel Check the logs for full command output."
我的requirements.txt是
ndjson==0.3.1
apache-beam[gcp]==2.26.0
dataclasses-json==0.5.2
google-cloud-storage==1.35.0
google-cloud-secret-manager==2.0.0
我正在使用以下 Dockerfile 创建 Flex 模板：
FROM gcr.io/dataflow-templates-base/python3-template-launcher-base
ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}
# Due to a change in the Beam base image in version 2.24, we need to install
# libffi-dev manually as a dependency. For more information:
# https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev git && rm -rf /var/lib/apt/lists/*
ADD utils utils
COPY requirements.txt .
COPY main.py .
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"
RUN pip install -U apache-beam[gcp]==2.24.0
RUN pip install -U -r ./requirements.txt
有人解决了这个问题吗？我确实尝试按照 中的建议设置特定的 setuptools 版本 (51.1.1)，导致出现相同的错误消息。有没有人遇到过同样的问题
问题是由于数据类中的冲突-json（我找不到确切原因）。从 requirements.txt 中删除后，可以成功构建图像：
ndjson==0.3.1
google-cloud-storage==1.31.2
google-cloud-secret-manager==2.0.0
pycryptodome==3.9.8
自今年年初以来，我们的 python 数据流作业导致 worker 启动时出错：
Processing /var/opt/google/dataflow/setuptools-51.1.1.tar.gz
ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
ERROR: No matching distribution found for wheel
ERROR: Command errored out with exit status 1: /usr/local/bin/python3 /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-rvfx0hn9/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=40.8.0' wheel Check the logs for full command output."
我的requirements.txt是
ndjson==0.3.1
apache-beam[gcp]==2.26.0
dataclasses-json==0.5.2
google-cloud-storage==1.35.0
google-cloud-secret-manager==2.0.0
我正在使用以下 Dockerfile 创建 Flex 模板：
FROM gcr.io/dataflow-templates-base/python3-template-launcher-base
ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}
# Due to a change in the Beam base image in version 2.24, we need to install
# libffi-dev manually as a dependency. For more information:
# https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev git && rm -rf /var/lib/apt/lists/*
ADD utils utils
COPY requirements.txt .
COPY main.py .
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"
RUN pip install -U apache-beam[gcp]==2.24.0
RUN pip install -U -r ./requirements.txt
有人解决了这个问题吗？我确实尝试按照
问题是由于数据类中的冲突-json（我找不到确切原因）。从 requirements.txt 中删除后，可以成功构建图像：
ndjson==0.3.1
google-cloud-storage==1.31.2
google-cloud-secret-manager==2.0.0
pycryptodome==3.9.8