如何通知 conda 安装更高版本的 apache-beam?

How do I inform conda to install a later version of apache-beam?

我是 Conda 新手,我正在尝试通过使用 miniconda 安装 python 软件包 apache-beam 来熟悉它。我可以在 https://anaconda.org/conda-forge/apache-beam 看到最新的可用版本是 v2.22.0

然而,当我尝试使用 conda install -c conda-forge/label/cf201901 apache-beam 安装时,它会尝试安装 v2.16.0:

# conda install -c conda-forge/label/cf201901 apache-beam
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: \
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - defaults/linux-64::mock==2.0.0=py37_0
  - defaults/linux-64::pyarrow==0.13.0=py37he6710b0_0
  - defaults/linux-64::mkl_random==1.1.1=py37h0573a6f_0
  - defaults/linux-64::arrow-cpp==0.13.0=py37h117bdfb_0
  - defaults/noarch::requests==2.25.1=pyhd3eb1b0_0
  - defaults/linux-64::cryptography==2.3.1=py37hc365091_0
  - defaults/noarch::packaging==20.8=pyhd3eb1b0_0
  - defaults/noarch::pytest-xdist==2.2.0=pyhd3eb1b0_0
  - conda-forge/linux-64::behave==1.2.6=py37h89c1867_1003
  - conda-forge/linux-64::parse_type==0.5.2=py37h89c1867_3
  - defaults/linux-64::pytest==6.2.1=py37h06a4308_0
  - defaults/noarch::python-dateutil==2.8.1=py_0
  - defaults/noarch::pytest-forked==1.3.0=py_0
  - defaults/linux-64::mkl-service==2.3.0=py37he8ac12f_0
  - defaults/linux-64::numpy-base==1.19.2=py37hfa32c7d_0
  - defaults/linux-64::pandas==1.1.5=py37ha9443f7_0
  - defaults/linux-64::requests-kerberos==0.12.0=py37_0
  - defaults/noarch::urllib3==1.26.2=pyhd3eb1b0_0
  - defaults/linux-64::numpy==1.19.2=py37h54aff64_0
  - defaults/linux-64::mkl_fft==1.2.0=py37h23d657b_0
  - defaults/noarch::python-hdfs==2.5.8=py_0
  - defaults/linux-64::pyopenssl==19.0.0=py37_0
done

## Package Plan ##

  environment location: /opt/conda/envs/python3

  added / updated specs:
    - apache-beam


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    apache-beam-2.16.0         |   py37h516909a_0         3.4 MB  conda-forge
    grpcio-1.16.0              |   py37hd60e7a3_0         1.0 MB  conda-forge/label/cf201901
    oauth2client-4.1.3         |             py_0          66 KB  conda-forge
    protobuf-3.6.0             |   py37hf484d3e_0         609 KB  main
    six-1.11.0                 |        py37_1001          21 KB  conda-forge/label/cf201901
    ------------------------------------------------------------
                                           Total:         5.1 MB

The following NEW packages will be INSTALLED:

  apache-beam        conda-forge/linux-64::apache-beam-2.16.0-py37h516909a_0
  grpcio             conda-forge/label/cf201901/linux-64::grpcio-1.16.0-py37hd60e7a3_0
  oauth2client       conda-forge/noarch::oauth2client-4.1.3-py_0
  protobuf           main/linux-64::protobuf-3.6.0-py37hf484d3e_0
  six                conda-forge/label/cf201901/linux-64::six-1.11.0-py37_1001

这是为什么?

您的命令无法为您提供最新版本的一个可能原因是,当您为 conda forge 指定 cf201901 标签时它不可用,您可以在 website:

而且当您尝试明确指定版本时:

$ conda install -c conda-forge/label/cf201901 apache-beam=2.22
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - apache-beam=2.22

Current channels:

  - https://conda.anaconda.org/conda-forge/label/cf201901/linux-64
  - https://conda.anaconda.org/conda-forge/label/cf201901/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

所以尽量省略标签并指定你想要的版本:

 conda install -c conda-forge apache-beam=2.22

不过我怀疑您已经将 conda-forge 添加到您的默认频道,因为您的输出已经显示 apache-beam 是从没有标签的 conda-forge 频道中提取的。

因此,如果以上仍然不起作用,则您当前环境中存在包(版本)冲突,conda 无法解决 apache-beam 版本 2.22

如果你只是为你需要的项目创建一个新的环境就可以解决这个问题apache-beam:

conda create -n <some name> -c conda-forge apache-beam

这将创建一个可以激活的环境

conda activate <some name>

@FlyingTeller 的回答准确地描述了问题和解决方案,但分享一些额外信息也无妨。

我忘了说我正在 docker 图像中构建我的 conda 环境,并且 docker 图像在 CI 管道中重建。为了确保我有一个可重复的构建,我在我的 repo 中有一个 environment.yaml 文件,它就是我用来构建 conda env 的文件。

在过去的一个月里,我不得不多次更改 environment.yaml 文件(我安装了 sh、apache-beam、pytest、pytest- xdict, ipython, behave...我认为这是我安装它们的顺序)我这样做的策略是:

  1. docker run myimage:most-recently-built-tag
  2. conda install 所需的包
  3. 导出新版本 environment.yaml
  4. 提交新版本 environment.yaml 到 repo

在这样做的过程中,我发现安装的 apache-beam 版本是 2.16.0,并且相互冲突的依赖项意味着我无法升级 apache-beam。

我创建了一个新环境并发布了 conda install sh apache-beam pytest pytest-xdict ipython behave,成功解决了所有依赖项并安装了 apache-beam=2.2.0。

因此我认为当我想对 environment.yaml 进行更改时,我的策略应该是:

  1. 创建一个新的 conda 环境
  2. conda install 一个命令中所有需要的包
  3. 导出一个新的 environment.yaml 并将其提交到存储库,以便它可以用于在 CI 中构建我的 docker 图像管道

欢迎对以上任何内容发表评论。


更新。这是我的 Dockerfile,它构建了一个 conda 环境,我可以从中导出 environment.yaml

FROM debian:buster-slim

RUN apt-get update && apt-get install curl gnupg -y

RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
        install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
        gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
        --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
        echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
        > /etc/apt/sources.list.d/conda.list

RUN MINICONDA_VERSION=4.9.2 && \
        CONDA_VERSION='4.9.*' && \
        CONDA_DIR=/opt/conda && \
        cd /tmp && \
        curl -O https://repo.anaconda.com/miniconda/Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh && \
        /bin/bash Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
        rm Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh && \
        $CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
        $CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
        $CONDA_DIR/bin/conda config --system --remove channels defaults && \
        $CONDA_DIR/bin/conda config --system --add channels main && \
        $CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
        $CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
        $CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
        $CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
        $CONDA_DIR/bin/conda config --system --append channels conda-forge  && \
        $CONDA_DIR/bin/conda create -n py3 python=3.7 

RUN bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install apache-beam sh pytest pytest-xdist ipython behave"

RUN echo "source /opt/conda/bin/activate /opt/conda/envs/py3" >> /root/.bashrc

并通过 运行:

生成我的 environment.yaml
docker build . -t conda-beam && \
  docker run --entrypoint bash conda-beam:latest -c \
  "source /opt/conda/bin/activate /opt/conda/envs/py3 \
  && conda env export"

我刚才运行的时候,返回

name: py3
channels:
  - main
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - apache-beam=2.22.0=py37h8f50634_0
  - apipkg=1.5=py37_0
  - arrow-cpp=0.15.1=py37h7cd5009_5
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=py_0
  - behave=1.2.6=py37h89c1867_1003
  - blas=1.0=mkl
  - boost-cpp=1.71.0=h7b6447c_0
  - brotli=1.0.9=he6710b0_2
  - brotlipy=0.7.0=py37h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2020.12.8=h06a4308_0
  - certifi=2020.12.5=py37h06a4308_0
  - cffi=1.14.4=py37h261ae71_0
  - chardet=4.0.0=py37h06a4308_1003
  - crcmod=1.7=py37hc8dfbb8_1003
  - cryptography=3.3.1=py37h3c74f83_0
  - cython=0.29.21=py37h2531618_0
  - decorator=4.4.2=py_0
  - dill=0.3.1.1=py37_1
  - docopt=0.6.2=py37_0
  - double-conversion=3.1.5=he6710b0_1
  - execnet=1.7.1=py_0
  - fastavro=0.23.5=py37h7b6447c_0
  - future=0.18.2=py37_1
  - gflags=2.2.2=he6710b0_0
  - glog=0.4.0=he6710b0_0
  - grpc-cpp=1.26.0=hf8bcb03_0
  - grpcio=1.31.0=py37hf8bcb03_0
  - httplib2=0.17.0=py37hc8dfbb8_1
  - icu=58.2=he6710b0_3
  - idna=2.10=py_0
  - importlib-metadata=2.0.0=py_1
  - importlib_metadata=2.0.0=1
  - iniconfig=1.1.1=py_0
  - intel-openmp=2020.2=254
  - ipython=7.19.0=py37hb070fc8_0
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - jedi=0.18.0=py37h06a4308_0
  - krb5=1.18.2=h173b8e3_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libboost=1.71.0=h97c9712_0
  - libedit=3.1.20191231=h14c3975_1
  - libevent=2.1.8=h1ba5d50_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libprotobuf=3.11.2=hd408876_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - lz4-c=1.8.1.2=h14c3975_0
  - mkl=2020.2=256
  - mkl-service=2.3.0=py37he8ac12f_0
  - mkl_fft=1.2.0=py37h23d657b_0
  - mkl_random=1.1.1=py37h0573a6f_0
  - mock=2.0.0=py37_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - ncurses=6.2=he6710b0_1
  - numpy=1.19.2=py37h54aff64_0
  - numpy-base=1.19.2=py37hfa32c7d_0
  - oauth2client=4.1.3=py_0
  - openssl=1.1.1i=h27cfd23_0
  - packaging=20.8=pyhd3eb1b0_0
  - pandas=1.2.0=py37ha9443f7_0
  - parse=1.18.0=pyh9f0ad1d_0
  - parse_type=0.5.2=py37h89c1867_3
  - parso=0.7.0=py_0
  - pbr=5.5.1=py_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pip=20.3.3=py37h06a4308_0
  - pluggy=0.13.1=py37_0
  - prompt-toolkit=3.0.8=py_0
  - protobuf=3.11.2=py37he6710b0_0
  - ptyprocess=0.7.0=pyhd3eb1b0_2
  - py=1.10.0=pyhd3eb1b0_0
  - pyarrow=0.15.1=py37h0573a6f_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pydot=1.3.0=py37_0
  - pygments=2.7.3=pyhd3eb1b0_0
  - pykerberos=1.2.1=py37h680d80a_2
  - pymongo=3.11.2=py37h2531618_0
  - pyopenssl=20.0.1=pyhd3eb1b0_1
  - pyparsing=2.4.7=py_0
  - pysocks=1.7.1=py37_1
  - pytest=6.2.1=py37h06a4308_0
  - pytest-forked=1.3.0=py_0
  - pytest-xdist=2.2.0=pyhd3eb1b0_0
  - python=3.7.9=h7579374_0
  - python-avro=1.9.2.1=py37hc8dfbb8_1
  - python-dateutil=2.8.1=py_0
  - python-hdfs=2.5.8=py_0
  - python_abi=3.7=1_cp37m
  - pytz=2020.5=pyhd3eb1b0_0
  - pyyaml=5.3.1=py37h7b6447c_1
  - re2=2020.11.01=h2531618_1
  - readline=8.0=h7b6447c_0
  - requests=2.25.1=pyhd3eb1b0_0
  - requests-kerberos=0.12.0=py37_0
  - rsa=4.7=pyhd3eb1b0_0
  - setuptools=51.1.2=py37h06a4308_3
  - sh=1.14.1=py37h06a4308_0
  - six=1.11.0=py37_1
  - snappy=1.1.8=he6710b0_0
  - sqlite=3.33.0=h62c20be_0
  - thrift-cpp=0.11.0=h02b749d_3
  - tk=8.6.10=hbc83047_0
  - toml=0.10.1=py_0
  - traitlets=5.0.5=py_0
  - typing-extensions=3.7.4.3=0
  - typing_extensions=3.7.4.3=py_0
  - uriparser=0.9.3=he6710b0_1
  - urllib3=1.26.2=pyhd3eb1b0_0
  - wcwidth=0.2.5=py_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - xz=5.2.5=h7b6447c_0
  - yaml=0.2.5=h7b6447c_0
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
prefix: /opt/conda/envs/py3