为什么搭建conda环境会版本冲突
Why version conflict when building conda environment
在我项目的 CI 管道中，我们构建了一个包含 conda 环境的 docker 图像。这是 Dockerfile：
FROM debian:buster-slim
RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
--fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
> /etc/apt/sources.list.d/conda.list
WORKDIR /tmp
RUN MINICONDA_VERSION=4.10.3 && \
MINICONDA_SCRIPT="Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" && \
CONDA_VERSION='4.10.3' && \
CONDA_DIR=/opt/conda && \
curl -O https://repo.anaconda.com/miniconda/${MINICONDA_SCRIPT} && \
/bin/bash ${MINICONDA_SCRIPT} -f -b -p $CONDA_DIR && \
rm ${MINICONDA_SCRIPT} && \
$CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
$CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
$CONDA_DIR/bin/conda config --system --remove channels defaults && \
$CONDA_DIR/bin/conda config --system --add channels main && \
$CONDA_DIR/bin/conda config --system --add channels conda-forge && \
$CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
$CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
$CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages conda-forge::google-* && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages ca-certificates && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages certifi && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages openssl && \
$CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
$CONDA_DIR/bin/conda create -n py3 python=3.8
RUN --mount=type=secret,id=pipconfig,dst=/etc/pip.conf \
bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
avro-python3 \
backports.zoneinfo \
behave \
black \
conda-tree \
dirty-equals \
flake8 \
google-api-python-client \
google-auth-oauthlib \
google-cloud-bigquery \
google-cloud-bigquery-storage \
google-cloud-bigtable \
google-cloud-datacatalog \
google-cloud-firestore \
google-cloud-logging \
google-cloud-monitoring \
google-cloud-pubsub \
google-cloud-secret-manager \
google-cloud-storage \
google-cloud-workflows \
invoke \
ipdb \
jinja2 \
jsonpath-ng \
oauth2client \
pandas \
pipdeptree \
pydantic \
pylint \
pytest \
pytest-xdist \
python-dateutil \
responses \
rich \
rope \
semantic_version \
sh \
sqlfluff \
tenacity \
tqdm \
zenpy \
&& pip install \
functions-framework \
google-cloud-bigquery==2.31.0 \
google-cloud-scheduler"
我想更新 conda 环境以使用 python 3.10 而不是 python3.8，所以我更改了相应的行：
$CONDA_DIR/bin/conda create -n py3 python=3.10
然后尝试构建 docker 图像。构建 conda 图像失败：
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- backports.zoneinfo -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0']
- google-cloud-bigquery -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
- google-cloud-bigquery-storage -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
- oauth2client -> python[version='2.7.*|3.5.*|3.6.*|3.4.*']
- python-dateutil -> python[version='>=3.8,<3.9.0a0|>=3.9,<3.10.0a0']
- tenacity -> python[version='3.4.*|3.8.*|3.7.*']
Your python: python=3.10
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
重要的一行似乎是：
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to.
我一直在深入研究以尝试发现问题并特别关注这一行：
google-cloud-bigquery -> python[version='2.7.|3.5.|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
该错误提示 google-cloud-bigquery 不适用于 python3.10，但是我已经在 https://anaconda.org/conda-forge/google-cloud-bigquery 进行了检查，其中指出该软件包应该适用于 python3.10
Description
google-cloud-bigquery installs google-cloud-bigquery-core and the extra requirements for pandas and pyarrow integrations.
Supported Python Versions
Python >= 3.6
The last version of this library compatible with Python 2.7 and 3.5 is google-cloud-bigquery==1.28.0.
在我看来，我尝试安装的软件包 是 可用于 python3.10，所以我不明白为什么我得到这个冲突。显然我遗漏了一些重要信息，但我对 conda 不够熟悉，无法理解什么。非常感谢任何建议。
我通过从 conda 安装中删除 backports.zoneinfo（在 python3.9 及更高版本中不需要，因为它被纳入标准库）和 google-cloud-bigquery 解决了这个问题。 google-cloud-bigquery 无论如何都被 pip 安装了（正如你在我原来的 post 中看到的那样），因为 conda 正在安装我们想要的版本的早期版本（我不知道为什么）。
我也删除了 tqdm，因为我们不再使用它了。生成的 Dockerfile 如下所示：
FROM debian:buster-slim
RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
--fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
> /etc/apt/sources.list.d/conda.list
WORKDIR /tmp
RUN MINICONDA_VERSION=4.10.3 && \
MINICONDA_SCRIPT="Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" && \
CONDA_VERSION='4.10.3' && \
CONDA_DIR=/opt/conda && \
curl -O https://repo.anaconda.com/miniconda/${MINICONDA_SCRIPT} && \
/bin/bash ${MINICONDA_SCRIPT} -f -b -p $CONDA_DIR && \
rm ${MINICONDA_SCRIPT} && \
$CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
$CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
$CONDA_DIR/bin/conda config --system --remove channels defaults && \
$CONDA_DIR/bin/conda config --system --add channels main && \
$CONDA_DIR/bin/conda config --system --add channels conda-forge && \
$CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
$CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
$CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages conda-forge::google-* && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages ca-certificates && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages certifi && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages openssl && \
$CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
$CONDA_DIR/bin/conda create -n py3 python=3.10
RUN --mount=type=secret,id=pipconfig,dst=/etc/pip.conf \
bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
avro-python3 \
behave \
black \
conda-tree \
dirty-equals \
flake8 \
google-api-python-client \
google-auth-oauthlib \
google-cloud-bigtable \
google-cloud-datacatalog \
google-cloud-firestore \
google-cloud-logging \
google-cloud-monitoring \
google-cloud-pubsub \
google-cloud-secret-manager \
google-cloud-storage \
google-cloud-workflows \
invoke \
ipdb \
jinja2 \
jsonpath-ng \
oauth2client \
pandas \
pipdeptree \
pydantic \
pylint \
pytest \
pytest-xdist \
python-dateutil \
responses \
rich \
rope \
semantic_version \
sh \
sqlfluff \
tenacity \
zenpy \
&& pip install \
functions-framework \
google-cloud-bigquery \
google-cloud-scheduler \
object-config-models"
在我项目的 CI 管道中，我们构建了一个包含 conda 环境的 docker 图像。这是 Dockerfile：
FROM debian:buster-slim
RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
--fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
> /etc/apt/sources.list.d/conda.list
WORKDIR /tmp
RUN MINICONDA_VERSION=4.10.3 && \
MINICONDA_SCRIPT="Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" && \
CONDA_VERSION='4.10.3' && \
CONDA_DIR=/opt/conda && \
curl -O https://repo.anaconda.com/miniconda/${MINICONDA_SCRIPT} && \
/bin/bash ${MINICONDA_SCRIPT} -f -b -p $CONDA_DIR && \
rm ${MINICONDA_SCRIPT} && \
$CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
$CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
$CONDA_DIR/bin/conda config --system --remove channels defaults && \
$CONDA_DIR/bin/conda config --system --add channels main && \
$CONDA_DIR/bin/conda config --system --add channels conda-forge && \
$CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
$CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
$CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages conda-forge::google-* && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages ca-certificates && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages certifi && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages openssl && \
$CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
$CONDA_DIR/bin/conda create -n py3 python=3.8
RUN --mount=type=secret,id=pipconfig,dst=/etc/pip.conf \
bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
avro-python3 \
backports.zoneinfo \
behave \
black \
conda-tree \
dirty-equals \
flake8 \
google-api-python-client \
google-auth-oauthlib \
google-cloud-bigquery \
google-cloud-bigquery-storage \
google-cloud-bigtable \
google-cloud-datacatalog \
google-cloud-firestore \
google-cloud-logging \
google-cloud-monitoring \
google-cloud-pubsub \
google-cloud-secret-manager \
google-cloud-storage \
google-cloud-workflows \
invoke \
ipdb \
jinja2 \
jsonpath-ng \
oauth2client \
pandas \
pipdeptree \
pydantic \
pylint \
pytest \
pytest-xdist \
python-dateutil \
responses \
rich \
rope \
semantic_version \
sh \
sqlfluff \
tenacity \
tqdm \
zenpy \
&& pip install \
functions-framework \
google-cloud-bigquery==2.31.0 \
google-cloud-scheduler"
我想更新 conda 环境以使用 python 3.10 而不是 python3.8，所以我更改了相应的行：
$CONDA_DIR/bin/conda create -n py3 python=3.10
然后尝试构建 docker 图像。构建 conda 图像失败：
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- backports.zoneinfo -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0']
- google-cloud-bigquery -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
- google-cloud-bigquery-storage -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
- oauth2client -> python[version='2.7.*|3.5.*|3.6.*|3.4.*']
- python-dateutil -> python[version='>=3.8,<3.9.0a0|>=3.9,<3.10.0a0']
- tenacity -> python[version='3.4.*|3.8.*|3.7.*']
Your python: python=3.10
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
重要的一行似乎是：
When python appears to the right, that indicates that the thing on the left is somehow not available for the python version you are constrained to.
我一直在深入研究以尝试发现问题并特别关注这一行：
google-cloud-bigquery -> python[version='2.7.|3.5.|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
该错误提示 google-cloud-bigquery 不适用于 python3.10，但是我已经在 https://anaconda.org/conda-forge/google-cloud-bigquery 进行了检查，其中指出该软件包应该适用于 python3.10
Description google-cloud-bigquery installs google-cloud-bigquery-core and the extra requirements for pandas and pyarrow integrations.
Supported Python Versions Python >= 3.6
The last version of this library compatible with Python 2.7 and 3.5 is google-cloud-bigquery==1.28.0.
在我看来，我尝试安装的软件包 是 可用于 python3.10，所以我不明白为什么我得到这个冲突。显然我遗漏了一些重要信息，但我对 conda 不够熟悉，无法理解什么。非常感谢任何建议。
我通过从 conda 安装中删除 backports.zoneinfo（在 python3.9 及更高版本中不需要，因为它被纳入标准库）和 google-cloud-bigquery 解决了这个问题。 google-cloud-bigquery 无论如何都被 pip 安装了（正如你在我原来的 post 中看到的那样），因为 conda 正在安装我们想要的版本的早期版本（我不知道为什么）。
我也删除了 tqdm，因为我们不再使用它了。生成的 Dockerfile 如下所示：
FROM debian:buster-slim
RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
--fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
> /etc/apt/sources.list.d/conda.list
WORKDIR /tmp
RUN MINICONDA_VERSION=4.10.3 && \
MINICONDA_SCRIPT="Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" && \
CONDA_VERSION='4.10.3' && \
CONDA_DIR=/opt/conda && \
curl -O https://repo.anaconda.com/miniconda/${MINICONDA_SCRIPT} && \
/bin/bash ${MINICONDA_SCRIPT} -f -b -p $CONDA_DIR && \
rm ${MINICONDA_SCRIPT} && \
$CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
$CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
$CONDA_DIR/bin/conda config --system --remove channels defaults && \
$CONDA_DIR/bin/conda config --system --add channels main && \
$CONDA_DIR/bin/conda config --system --add channels conda-forge && \
$CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
$CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
$CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages conda-forge::google-* && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages ca-certificates && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages certifi && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages openssl && \
$CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
$CONDA_DIR/bin/conda create -n py3 python=3.10
RUN --mount=type=secret,id=pipconfig,dst=/etc/pip.conf \
bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
avro-python3 \
behave \
black \
conda-tree \
dirty-equals \
flake8 \
google-api-python-client \
google-auth-oauthlib \
google-cloud-bigtable \
google-cloud-datacatalog \
google-cloud-firestore \
google-cloud-logging \
google-cloud-monitoring \
google-cloud-pubsub \
google-cloud-secret-manager \
google-cloud-storage \
google-cloud-workflows \
invoke \
ipdb \
jinja2 \
jsonpath-ng \
oauth2client \
pandas \
pipdeptree \
pydantic \
pylint \
pytest \
pytest-xdist \
python-dateutil \
responses \
rich \
rope \
semantic_version \
sh \
sqlfluff \
tenacity \
zenpy \
&& pip install \
functions-framework \
google-cloud-bigquery \
google-cloud-scheduler \
object-config-models"