Github 操作 CI：让 PyCall 知道 Julia ENV 变量

Question

我想设置一个 Github 操作 ci.yml 文件来安装 Python 和 Julia，然后使用 PyCall 构建和测试包。在这种情况下，PyCall 应该使用先前安装的 Python 版本，而不是它的默认 julia-specific Python 版本，它带有它的 miniconda。

到目前为止，我的 ci.yml 看起来如下所示：

name: CI
on:
  - push
  - pull_request

test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        version:
          - '1.5'
          - 'nightly'
        python: [3.7]
        # python-path: [RUNNER_TOOL_CACHE/Python/*]
        os:
          - macOS-latest
         
        arch:
          - x64
steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python }}
      - name: Display Python version
        run: python -c "import sys; print(sys.version)"

- name: Install dependencies 
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

# Julia Tasks
- uses: julia-actions/setup-julia@v1
- name: Setup Julia
  with:
    version: ${{ matrix.version }}
    arch: ${{ matrix.arch }}

- name: Set ENV variables for  
        run: |
          echo ENV["PYTHON"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME_PYTHONHOME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV

- name: Build Julia pkg
        run: |
          echo $ENV["PYTHON"]
          echo $ENV["PYCALL_JL_RUNTIME_PYTHONHOME"]
          echo $ENV["PYCALL_JL_RUNTIME"]
      - uses: julia-actions/julia-buildpkg@v1
              
      - uses: julia-actions/julia-runtest@v1
      - name: Test Julia

它运行在 julia-actions/julia-runtest@v1 之前都很好，到目前为止原因很清楚，因为 PyCall 使用它自己的 miniconda 环境和 Python 安装以及 python-package我尝试在我的pkg.jlwith PyCall中加载，安装在系统Python我运行的CIwith.

我不想使用内部 PyCall -Julia specific- Python 版本，因为我希望能够通过 Matrix 调用测试不同的 Python 版本。

我可以通过 Github 操作建议的方式传递环境变量，例如：

- name: Set ENV Variables for  
        run: |
          echo ENV["PYTHON"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME_PYTHONHOME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV

输出：

Run echo $ENV["PYTHON"]
  echo $ENV["PYTHON"]
  echo $ENV["PYCALL_JL_RUNTIME_PYTHONHOME"]
  echo $ENV["PYCALL_JL_RUNTIME"]
  shell: /bin/bash -e {0}
  env:
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYTHON] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYCALL_JL_RUNTIME_PYTHONHOME] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYCALL_JL_RUNTIME] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
[PYTHON]
[PYCALL_JL_RUNTIME_PYTHONHOME]
[PYCALL_JL_RUNTIME]

这是 PyCall Documentation

建议的

但是，PyCall 仍然使用：

┌ Info: Using the Python distribution in the Conda package by default.
└ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
[ Info: Running `conda install -q -y numpy` in root environment
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

[ Info: PyCall is using /Users/runner/.julia/conda/3/bin/python (Python 3.8.5) at /Users/runner/.julia/conda/3/bin/python, libpython = /Users/runner/.julia/conda/3/lib/libpython3.8.dylib
[ Info: /Users/runner/.julia/packages/PyCall/BcTLp/deps/deps.jl has not changed
[ Info: /Users/runner/.julia/prefs/PyCall has not changed

我可以在 julia-actions/setup-julia@v1 中传递一个 startup.jl 文件吗？

如何让 PyCall 知道这些 ENV 变量？

Edit: Added env to julia-actions/julia-buildpkg@v1 and rebuilt PyCall

它给我一个权限错误：

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : ${{ env.pythonLocation }}
          PYCALL_JL_RUNTIME_PYTHONHOME: ${{ env.pythonLocation }}
          PYCALL_JL_RUNTIME: ${{ env.pythonLocation }}
      - run: |
          julia 
          using Pkg
          Pkg.build("PyCall")

     Building PyCall → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/b6dff5fa725eff4f775f472acd86756d6e31fb02/build.log`
┌ Info: No system-wide Python was found; got the following error:
│ Base.IOError("could not spawn setenv(`/Users/runner/hostedtoolcache/Python/3.7.9/x64 -c \"import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))\"`,[\"PATH=/Users/runner/hostedtoolcache/julia/nightly/x64/bin:/Users/runner/hostedtoolcache/Python/3.7.9/x64/bin:/Users/runner/hostedtoolcache/Python/3.7.9/x64:/usr/local/opt/pipx_bin:/Users/runner/.cargo/bin:/usr/local/lib/ruby/gems/2.7.0/bin:/usr/local/opt/ruby/bin:/usr/local/opt/curl/bin:/usr/local/bin:/usr/local/sbin:/Users/runner/bin:/Users/runner/.yarn/bin:/usr/local/go/bin:/Users/runner/Library/Android/sdk/tools:/Users/runner/Library/Android/sdk/platform-tools:/Users/runner/Library/Android/sdk/ndk-bundle:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/sbin:/Users/runner/.dotnet/tools:/Users/runner/.ghcup/bin:/Users/runner/hostedtoolcache/stack/2.5.1/x64\", \"GITHUB_RUN_NUMBER=90\", \"PERFLOG_LOCATION_SETTING=RUNNER_PERFLOG\", \"NUNIT3_PATH=/Library/Developer/nunit/3.6.0\", \"NVM_CD_FLAGS=\", \"GITHUB_ACTOR=mkschulze\", \"ANDROID_NDK_18R_PATH=/Users/runner/Library/Android/sdk/ndk/18.1.5063045\", \"ENV[PYCALL_JL_RUNTIME_PYTHONHOME] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"USER=runner\", \"PYCALL_JL_RUNTIME_PYTHONHOME=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"CONDA=/usr/local/miniconda\", \"VCPKG_INSTALLATION_ROOT=/usr/local/share/vcpkg\", \"GITHUB_SHA=a08b0b9021e3138eb34ff298f2ccc0498713e627\", \"JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home\", \"GITHUB_API_URL=https://api.github.com\", \"GITHUB_ACTIONS=true\", \"ANDROID_SDK_ROOT=/Users/runner/Library/Android/sdk\", \"GITHUB_ENV=/Users/runner/work/_temp/_runner_file_commands/set_env_e539860f-4c69-4d29-9773-3ad63e77ebdf\", \"SHELL=/bin/bash\", \"GITHUB_ACTION_PATH=/Users/runner/work/_actions/julia-actions/julia-buildpkg/v1\", \"RUNNER_PERFLOG=/usr/local/opt/runner/perflog\", \"JAVA_HOME_12_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-12.jdk/Contents/Home\", \"EDGEWEBDRIVER=/usr/local/share/edge_driver\", \"TMPDIR=/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/\", \"GITHUB_RUN_ID=390315562\", \"ImageOS=macos1015\", \"NVM_DIR=/Users/runner/.nvm\", \"GITHUB_HEAD_REF=\", \"GITHUB_RETENTION_DAYS=90\", \"GITHUB_SERVER_URL=https://github.com\", \"GITHUB_JOB=test\", \"RUNNER_TOOL_CACHE=/Users/runner/hostedtoolcache\", \"RUNNER_TRACKING_ID=github_bab8cac0-d4b7-441b-9529-36f2faf19750\", \"HOMEBREW_NO_AUTO_UPDATE=1\", \"CHROMEWEBDRIVER=/usr/local/Caskroom/chromedriver/86.0.4240.22\", \"GITHUB_WORKFLOW=CI\", \"GITHUB_ACTION_REPOSITORY=julia-actions/julia-buildpkg\", \"PYCALL_JL_RUNTIME=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"pythonLocation=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"HOME=/Users/runner\", \"GITHUB_ACTION=julia-actionsjulia-buildpkg\", \"ENV[PYTHON] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"PYTHONIOENCODING=UTF-8\", \"JAVA_HOME_8_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home\", \"GITHUB_EVENT_PATH=/Users/runner/work/_temp/_github_workflow/event.json\", \"XPC_FLAGS=0x0\", \"PYTHON=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"XCODE_10_DEVELOPER_DIR=/Applications/Xcode_10.3.app/Contents/Developer\", \"XPC_SERVICE_NAME=0\", \"XCODE_12_DEVELOPER_DIR=/Applications/Xcode_12.2.app/Contents/Developer\", \"GITHUB_REF=refs/heads/py-cli\", \"XCODE_11_DEVELOPER_DIR=/Applications/Xcode_11.7.app/Contents/Developer\", \"__CF_USER_TEXT_ENCODING=0x1F5:0:0\", \"LC_ALL=en_US.UTF-8\", \"GITHUB_REPOSITORY=Humans-of-Julia/GraknClient.jl\", \"LC_CTYPE=en_US.UTF-8\", \"ImageVersion=20201115.1\", \"LANG=en_US.UTF-8\", \"GITHUB_GRAPHQL_URL=https://api.github.com/graphql\", \"ENV[PYCALL_JL_RUNTIME] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"SHLVL=2\", \"LOGNAME=runner\", \"RUNNER_WORKSPACE=/Users/runner/work/GraknClient.jl\", \"SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.yxv10u7SRe/Listeners\", \"DOTNET_MULTILEVEL_LOOKUP=0\", \"HOMEBREW_CASK_OPTS=--no-quarantine\", \"GITHUB_BASE_REF=\", \"_=/Users/runner/hostedtoolcache/julia/nightly/x64/bin/julia\", \"GITHUB_ACTION_REF=v1\", \"DOTNET_ROOT=/Users/runner/.dotnet\", \"PIPX_HOME=/usr/local/opt/pipx\", \"***", \"JAVA_HOME_13_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-13.jdk/Contents/Home\", \"GITHUB_WORKSPACE=/Users/runner/work/GraknClient.jl/GraknClient.jl\", \"NUNIT_BASE_PATH=/Library/Developer/nunit\", \"ANDROID_HOME=/Users/runner/Library/Android/sdk\", \"CI=true\", \"POWERSHELL_DISTRIBUTION_CHANNEL=GitHub-Actions-macos1015\", \"GECKOWEBDRIVER=/usr/local/opt/geckodriver/bin\", \"RCT_NO_LAUNCH_PACKAGER=1\", \"GITHUB_PATH=/Users/runner/work/_temp/_runner_file_commands/add_path_e539860f-4c69-4d29-9773-3ad63e77ebdf\", \"RUNNER_OS=macOS\", \"JAVA_HOME_7_X64=/Library/Java/JavaVirtualMachines/zulu-7.jdk/Contents/Home\", \"JULIA_LOAD_PATH=@:/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/jl_XGe9fC\", \"PIPX_BIN_DIR=/usr/local/opt/pipx_bin\", \"BOOTSTRAP_HASKELL_NONINTERACTIVE=1\", \"JAVA_HOME_14_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-14.jdk/Contents/Home\", \"JAVA_HOME_11_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home\", \"RUNNER_TEMP=/Users/runner/work/_temp\", \"GITHUB_REPOSITORY_OWNER=Humans-of-Julia\", \"GITHUB_EVENT_NAME=push\", \"OPENBLAS_MAIN_FREE=1\", \"ANDROID_NDK_HOME=/Users/runner/Library/Android/sdk/ndk-bundle\", \"AGENT_TOOLSDIRECTORY=/Users/runner/hostedtoolcache\"])
: permission denied (EACCES)", -13)
└ using the Python distribution in the Conda package
[ Info: Running `conda install -q -y numpy` in root environment
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

Answer 1

每次更改 ENV["PYTHON"] 后，您都需要运行 Pkg.build("PyCall")。这是 Julia 真正读取 Python 的配置的唯一时刻 - 否则它根本不在乎。

如果你想测试一个 Python 配置的矩阵，我仍然建议你为每个配置一个 miniconda Python （尽管它不需要由 Julia 安装 - 它可以在 Julia 外部使用 miniconda）。当运行ning Pkg.build("PyCall") 使用非 Conda Python 时，各种“问题”往往会出现，您可能最终会为每个安装微调 Julia。

Answer 2

好的，现在知道了运行。将此 PYTHON ENV 设置为确切的 Python 二进制文件，至少在 macOS 和 Ubuntu x64 上它可以工作。

macOS / linux解决方案：

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/bin/python"

Windows 解法：

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/python.exe"

所有人的解决方案：

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : python

这是在 macOS 和 Ubuntu x64 上传递的完整工作 ci.yml：

name: CI
on:
  - push
  - pull_request

jobs:
  test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        version:
          - '1.5'
          - 'nightly'
        python: [3.7]
        os:
          - ubuntu-latest
          - macOS-latest
        arch:
          - x64
    steps:
      - uses: actions/checkout@v2
      - name: Set up  ${{ matrix.python }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python }}

      - name: Display Python version
        run: python -c "import sys; print(sys.version)"

      - name: Set ENV Variables for  
        run: echo ENV["PYTHON"] = "${{ env.pythonLocation }}/bin/python" >> $GITHUB_ENV

      - name: Install dependencies 
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      # Julia Tasks
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
        
      - uses: actions/cache@v2
        env:
          cache-name: cache-artifacts
        with:
          path: ~/.julia/artifacts
          key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
          restore-keys: |
            ${{ runner.os }}-test-${{ env.cache-name }}-
            ${{ runner.os }}-test-
            ${{ runner.os }}-

      
      - uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/bin/python"
            
      - uses: julia-actions/julia-runtest@v1
      - uses: julia-actions/julia-processcoverage@v1
      - uses: codecov/codecov-action@v1
        with:
          file: lcov.info

Github 操作 CI：让 PyCall 知道 Julia ENV 变量

Github Actions CI: making PyCall aware of Julia ENV variables

continuous-integration

julia

github-actions

pycall

Github 操作 CI：让 PyC​​all 知道 Julia ENV 变量

Github Actions CI: making PyCall aware of Julia ENV variables

continuous-integration

julia

github-actions

pycall

Github 操作 CI：让 PyCall 知道 Julia ENV 变量