使用特定 Miniconda Python 和 NumPy 版本进行 CircleCI 测试
CircleCI testing with specific Miniconda Python and NumPy versions
我正在做一个使用 CircleCI 持续集成平台的项目。我使用 Python 作为主要语言，使用 Miniconda 作为平台。我想在 CircleCI 上使用 Miniconda 测试多个 Python 和 NumPy 版本。
我尝试使用不同的 Python 图像，但它只使用 Python 3.7，因为我安装了最新的 Miniconda 版本。你能告诉我如何使用多个版本吗？
下面是config.yml:
version: 2.0
workflows:
version: 2
test:
jobs:
- py3.6-np1.15
- py3.5-np1.15
- py3.6-np1.14
- py3.5-np1.14
- py3.7-np1.15
- py3.5-np1.16
- py3.6-np1.16
- py3.7-np1.16
jobs:
py3.6-np1.15: &test-template
docker:
- image: circleci/python:3.6.8
environment:
NUMPY_VERSION: 1.15.2
CYTHON_VERSION: 0.29.2
working_directory: ~/repo
steps:
- checkout
- run:
name: Install System Dependencies
command: sudo apt-get update && sudo apt-get install -y libmpich12 libmpich-dev build-essential
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum "setup.py" }}
- run:
name: install anaconda
command: |
wget https://repo.continuum.io/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh -O ~/miniconda.sh
chmod +x ~/miniconda.sh && ~/miniconda.sh -b
export PATH=$HOME/miniconda3/bin:$PATH
conda update --quiet --yes conda
- run:
name: Install numpy, cython, mdtraj
command: |
export PATH=$HOME/miniconda3/bin:$PATH
conda install --quiet --yes numpy==$NUMPY_VERSION cython==$CYTHON_VERSION
conda install --quiet --yes -c conda-forge mdtraj
# - run:
# name: Upgrade pip
# command: |
# python3 -m venv venv
# . venv/bin/activate
# pip install pip==18.0
# - run:
# name: Install numpy and cython
# command: |
# python3 -m venv venv
# . venv/bin/activate
# pip install --progress-bar off numpy==$NUMPY_VERSION cython==$CYTHON_VERSION
- run:
name: Install and build
command: |
export PATH=$HOME/miniconda3/bin:$PATH
pip install --progress-bar off .[dev]
python setup.py build_ext --inplace
python setup.py install
py3.5-np1.15:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.6-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.5-np1.14:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.7-np1.15:
<<: *test-template
docker:
- image: circleci/python:3.7.3
py3.5-np1.16:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
py3.6-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
py3.7-np1.16:
<<: *test-template
docker:
- image: circleci/python:3.7.3
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
这里是关于如何将 CircleCI 与 Miniconda 和特定 Python 以及 NumPy 版本一起使用的最小示例配置，从一个空的 ubuntu:bionic 图像开始。
version: 2
jobs:
build:
docker:
- image: ubuntu:bionic
environment:
PYTHON_VERSION: 3.5.5
NUMPY_VERSION: 1.14.2
steps:
- checkout
- run:
name: Setup Miniconda
command: |
apt update
apt install -y wget
cd $HOME
wget "https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh" -O miniconda.sh
printf '%s' "8a324adcc9eaf1c09e22a992bb6234d91a94146840ee6b11c114ecadafc68121 miniconda.sh" | sha256sum -c
bash miniconda.sh -b -p $HOME/miniconda
- run:
name: Setup environment and run tests
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update -y conda
conda create -n myenv python=$PYTHON_VERSION -c conda-forge
source activate myenv
conda install -y numpy=$NUMPY_VERSION
python --version
python -c "import numpy; print(numpy.__version__)"
我认为从 Internet 下载 Miniconda 安装脚本 Miniconda3-4.7.10-Linux-x86_64.sh 后验证校验和是一个很好的做法。
您可以更改环境变量 PYTHON_VERSION 和 NUMPY_VERSION 以获得其他版本。
而不是 "real" 测试，目前我们只是要验证我们想要的 Python 和 NumPy 版本是否正在与 python --version 和 python -c "import numpy; print(numpy.__version__)" 一起使用。对于上面的示例，在日志的末尾您应该找到：
Python 3.5.5
1.14.2
根据您 select 的版本，您可能会遇到错误：
- 如果你得到
PackagesNotFoundError，你需要确保 selected 频道有你要找的包版本。 （例如 conda-forge 在上面的例子中被 select 编辑。）
- 如果你得到
UnsatisfiableError，你有 select 版本的包不兼容。
这是多个版本的示例配置：
version: 2
workflows:
version: 2
test:
jobs:
- python_3.5
- python_3.6
- python_3.7
template: &template
docker:
- image: ubuntu:bionic
steps:
- checkout
- run:
name: Setup Miniconda
command: |
apt update
apt install -y wget
cd $HOME
wget "https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh" -O miniconda.sh
printf '%s' "8a324adcc9eaf1c09e22a992bb6234d91a94146840ee6b11c114ecadafc68121 miniconda.sh" | sha256sum -c
bash miniconda.sh -b -p $HOME/miniconda
- run:
name: Setup environment and run tests
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update -y conda
conda create -n myenv python=$PYTHON_VERSION
source activate myenv
conda install -y pip numpy=$NUMPY_VERSION
python --version
pip --version
python -c "import numpy; print(numpy.__version__)"
jobs:
python_3.5:
<<: *template
environment:
PYTHON_VERSION: 3.5
NUMPY_VERSION: 1.14.2
python_3.6:
<<: *template
environment:
PYTHON_VERSION: 3.6
NUMPY_VERSION: 1.15.2
python_3.7:
<<: *template
environment:
PYTHON_VERSION: 3.7
NUMPY_VERSION: 1.16.5
如果我将这个最小示例应用到您的案例中，配置将类似于：
version: 2.0
workflows:
version: 2
test:
jobs:
- py3.6-np1.15
- py3.5-np1.15
- py3.6-np1.14
- py3.5-np1.14
- py3.7-np1.15
- py3.6-np1.16
- py3.7-np1.16
test-template: &test-template
docker:
- image: ubuntu:bionic
steps:
- checkout
- run:
name: Install System Dependencies
command: apt-get update && apt-get install -y libmpich12 libmpich-dev build-essential
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum "setup.py" }}
- run:
name: install anaconda
command: |
apt update
apt install -y wget
cd $HOME
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
chmod +x ~/miniconda.sh && bash ~/miniconda.sh -b -p $HOME/miniconda
export PATH=$HOME/miniconda/bin:$PATH
- run:
name: Install numpy, cython, mdtraj
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update --yes conda
echo $PYTHON_VERSION
conda create -n myenv python=$PYTHON_VERSION -c conda-forge
source activate myenv
conda install --yes pip
conda install --yes -c conda-forge numpy=$NUMPY_VERSION cython=$CYTHON_VERSION
conda install --yes -c conda-forge nose mdtraj
python --version
python -c "import numpy; print(numpy.__version__)"
- run:
name: Install and build package
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
pip install --progress-bar off .[dev]
python setup.py build_ext --inplace
python setup.py install
- save_cache:
paths:
- ~/miniconda
key: v1-dependencies-{{ checksum "setup.py" }}
- run:
name: Run non-MPI tests
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
nosetests -a '!mpi' package
- run:
name: Run MPI tests
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
OMP_NUM_THREADS=1 mpiexec -n 2 nosetests -a mpi package
- store_artifacts:
path: test-reports
destination: test-reports
jobs:
py3.6-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.5-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.5
py3.6-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.5-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.5
py3.7-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.15.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.7.1
py3.6-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.7-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
PYTHON_VERSION: 3.7.1
我正在做一个使用 CircleCI 持续集成平台的项目。我使用 Python 作为主要语言，使用 Miniconda 作为平台。我想在 CircleCI 上使用 Miniconda 测试多个 Python 和 NumPy 版本。
我尝试使用不同的 Python 图像，但它只使用 Python 3.7，因为我安装了最新的 Miniconda 版本。你能告诉我如何使用多个版本吗？
下面是config.yml:
version: 2.0
workflows:
version: 2
test:
jobs:
- py3.6-np1.15
- py3.5-np1.15
- py3.6-np1.14
- py3.5-np1.14
- py3.7-np1.15
- py3.5-np1.16
- py3.6-np1.16
- py3.7-np1.16
jobs:
py3.6-np1.15: &test-template
docker:
- image: circleci/python:3.6.8
environment:
NUMPY_VERSION: 1.15.2
CYTHON_VERSION: 0.29.2
working_directory: ~/repo
steps:
- checkout
- run:
name: Install System Dependencies
command: sudo apt-get update && sudo apt-get install -y libmpich12 libmpich-dev build-essential
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum "setup.py" }}
- run:
name: install anaconda
command: |
wget https://repo.continuum.io/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh -O ~/miniconda.sh
chmod +x ~/miniconda.sh && ~/miniconda.sh -b
export PATH=$HOME/miniconda3/bin:$PATH
conda update --quiet --yes conda
- run:
name: Install numpy, cython, mdtraj
command: |
export PATH=$HOME/miniconda3/bin:$PATH
conda install --quiet --yes numpy==$NUMPY_VERSION cython==$CYTHON_VERSION
conda install --quiet --yes -c conda-forge mdtraj
# - run:
# name: Upgrade pip
# command: |
# python3 -m venv venv
# . venv/bin/activate
# pip install pip==18.0
# - run:
# name: Install numpy and cython
# command: |
# python3 -m venv venv
# . venv/bin/activate
# pip install --progress-bar off numpy==$NUMPY_VERSION cython==$CYTHON_VERSION
- run:
name: Install and build
command: |
export PATH=$HOME/miniconda3/bin:$PATH
pip install --progress-bar off .[dev]
python setup.py build_ext --inplace
python setup.py install
py3.5-np1.15:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.6-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.5-np1.14:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.7-np1.15:
<<: *test-template
docker:
- image: circleci/python:3.7.3
py3.5-np1.16:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
py3.6-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
py3.7-np1.16:
<<: *test-template
docker:
- image: circleci/python:3.7.3
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
这里是关于如何将 CircleCI 与 Miniconda 和特定 Python 以及 NumPy 版本一起使用的最小示例配置，从一个空的 ubuntu:bionic 图像开始。
version: 2
jobs:
build:
docker:
- image: ubuntu:bionic
environment:
PYTHON_VERSION: 3.5.5
NUMPY_VERSION: 1.14.2
steps:
- checkout
- run:
name: Setup Miniconda
command: |
apt update
apt install -y wget
cd $HOME
wget "https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh" -O miniconda.sh
printf '%s' "8a324adcc9eaf1c09e22a992bb6234d91a94146840ee6b11c114ecadafc68121 miniconda.sh" | sha256sum -c
bash miniconda.sh -b -p $HOME/miniconda
- run:
name: Setup environment and run tests
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update -y conda
conda create -n myenv python=$PYTHON_VERSION -c conda-forge
source activate myenv
conda install -y numpy=$NUMPY_VERSION
python --version
python -c "import numpy; print(numpy.__version__)"
我认为从 Internet 下载 Miniconda 安装脚本 Miniconda3-4.7.10-Linux-x86_64.sh 后验证校验和是一个很好的做法。
您可以更改环境变量 PYTHON_VERSION 和 NUMPY_VERSION 以获得其他版本。
而不是 "real" 测试，目前我们只是要验证我们想要的 Python 和 NumPy 版本是否正在与 python --version 和 python -c "import numpy; print(numpy.__version__)" 一起使用。对于上面的示例，在日志的末尾您应该找到：
Python 3.5.5
1.14.2
根据您 select 的版本，您可能会遇到错误：
- 如果你得到
PackagesNotFoundError，你需要确保 selected 频道有你要找的包版本。 （例如conda-forge在上面的例子中被 select 编辑。） - 如果你得到
UnsatisfiableError，你有 select 版本的包不兼容。
这是多个版本的示例配置：
version: 2
workflows:
version: 2
test:
jobs:
- python_3.5
- python_3.6
- python_3.7
template: &template
docker:
- image: ubuntu:bionic
steps:
- checkout
- run:
name: Setup Miniconda
command: |
apt update
apt install -y wget
cd $HOME
wget "https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh" -O miniconda.sh
printf '%s' "8a324adcc9eaf1c09e22a992bb6234d91a94146840ee6b11c114ecadafc68121 miniconda.sh" | sha256sum -c
bash miniconda.sh -b -p $HOME/miniconda
- run:
name: Setup environment and run tests
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update -y conda
conda create -n myenv python=$PYTHON_VERSION
source activate myenv
conda install -y pip numpy=$NUMPY_VERSION
python --version
pip --version
python -c "import numpy; print(numpy.__version__)"
jobs:
python_3.5:
<<: *template
environment:
PYTHON_VERSION: 3.5
NUMPY_VERSION: 1.14.2
python_3.6:
<<: *template
environment:
PYTHON_VERSION: 3.6
NUMPY_VERSION: 1.15.2
python_3.7:
<<: *template
environment:
PYTHON_VERSION: 3.7
NUMPY_VERSION: 1.16.5
如果我将这个最小示例应用到您的案例中，配置将类似于：
version: 2.0
workflows:
version: 2
test:
jobs:
- py3.6-np1.15
- py3.5-np1.15
- py3.6-np1.14
- py3.5-np1.14
- py3.7-np1.15
- py3.6-np1.16
- py3.7-np1.16
test-template: &test-template
docker:
- image: ubuntu:bionic
steps:
- checkout
- run:
name: Install System Dependencies
command: apt-get update && apt-get install -y libmpich12 libmpich-dev build-essential
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum "setup.py" }}
- run:
name: install anaconda
command: |
apt update
apt install -y wget
cd $HOME
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
chmod +x ~/miniconda.sh && bash ~/miniconda.sh -b -p $HOME/miniconda
export PATH=$HOME/miniconda/bin:$PATH
- run:
name: Install numpy, cython, mdtraj
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update --yes conda
echo $PYTHON_VERSION
conda create -n myenv python=$PYTHON_VERSION -c conda-forge
source activate myenv
conda install --yes pip
conda install --yes -c conda-forge numpy=$NUMPY_VERSION cython=$CYTHON_VERSION
conda install --yes -c conda-forge nose mdtraj
python --version
python -c "import numpy; print(numpy.__version__)"
- run:
name: Install and build package
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
pip install --progress-bar off .[dev]
python setup.py build_ext --inplace
python setup.py install
- save_cache:
paths:
- ~/miniconda
key: v1-dependencies-{{ checksum "setup.py" }}
- run:
name: Run non-MPI tests
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
nosetests -a '!mpi' package
- run:
name: Run MPI tests
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
OMP_NUM_THREADS=1 mpiexec -n 2 nosetests -a mpi package
- store_artifacts:
path: test-reports
destination: test-reports
jobs:
py3.6-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.5-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.5
py3.6-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.5-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.5
py3.7-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.15.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.7.1
py3.6-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.7-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
PYTHON_VERSION: 3.7.1