包设置不会传播给分布式工作人员

Question

信息：

$ julia --version
julia version 1.6.0
$ lscpu
~/root/MyPackage$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           158
Model name:                      Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
...

假设我想要以下包结构，并想使用 ReTest 的并行测试（我的问题似乎与代码加载在分布式中的工作方式有关，所以这并不是真正的 ReTest 特定问题）。

| root/
    | MyPackage/
        | Project.toml
        | Manifest.toml
        | src/
            | MyPackage.jl
        | test/
            | runtests.jl
            | MyPackageTests.jl

我按以下方式初始化了这个包：

$ cd root && julia
(...) pkg> generate MyPackage;
$ cd MyPackage && julia
(...) pkg> activate .
(...) pkg> instantiate
(...) pkg> add ReTest InlineTest Distributed;
...

用一些 Julia 代码填写 MyPackage.jl、runtests.jl 和 MyPackageTests.jl。该代码是什么不太重要 - 尽管我遵循 here in ReTest.

的指南

然后设置：

$ julia
(...) pkg> activate .
(...) pkg> instantiate
(MyPackage) pkg> st
     Project MyPackage v0.1.0
      Status `~/root/MyPackage/Project.toml`
  [bd334432] InlineTest v0.2.0
  [e0db7c4e] ReTest v0.3.2
  [8ba89e20] Distributed
julia> LOAD_PATH
3-element Vector{String}:
 "@"        # Should be current active environment for MyPackage
 "@v#.#"    # Should be @v1.6 on my system
 "@stdlib"  # Should be absolute path of current Julia installation's stdlib
julia> # Should this code be in .jl files? Don't think that should matter.
julia> using Distributed
julia> addprocs(2)
julia> @everywhere include("test/MyPackageTests.jl")
ERROR: On worker 2:
LoadError: ArgumentError: Package MyPackage not found in current path:
- Run `import Pkg; Pkg.add("MyPackage")` to install the MyPackage package.

Stacktrace:
 [1] require
   @ ./loading.jl:871
 [2] include
   @ ./client.jl:444
 [3] top-level scope
   @ none:1
 [4] eval
   @ ./boot.jl:360
 [5] #103
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
 [6] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
 [7] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
 [8] #96
   @ ./task.jl:406
in expression starting at /path/to/root/MyPackage/test/MyPackageTests.jl:1

...and 2 more exceptions.

Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:364
 [2] macro expansion
   @ ./task.jl:383 [inlined]
 [3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr)
   @ Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:223
 [4] top-level scope
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:207

我想了解这是为什么。在 GitHub 我看到 this issue that was supposedly fixed。我可以确认当我运行 that 示例时，如果 using 语句涉及环境所在的包，我也会遇到与上面的 MyPackage 完全相同的问题对于.

在提交错误或打开问题之前，我想在这里检查一下，以防这是我的流程问题。如果不是，那么 Distributed/ReTest 显然有问题，我会为那些问题开票。非常感谢任何帮助。

Answer 1

试试下面的代码：

using Distributed
addprocs(4) # or whatever you need or use the -p parameter
using Pkg
pkg"activate ."
pkg"instantiate"  # run this when needed
using MyPackage # first load package only an the master worker

@everywhere pkg"activate ."
@everywhere using MyPackage

说明：每个 Julia 进程都是完全独立的，因此它有自己的包状态、变量、内存等。

请注意，您通常更愿意先在主节点上加载包，因为某些包可能会在首次加载时执行某些操作。

Answer 2

根据@carstenbauer 的说法，默认情况下，活动的 Julia 环境不会自动传播到工作进程。解决这个问题的方法是在调用 addprocs 的参数中设置环境，如下所示：

julia> using Distributed
julia> addprocs(2, exeflags="--project=$(Base.active_project())")
julia> @everywhere include("test/MyPackageTests.jl")
julia> MyPackageTests.runtests()  # runs to completion

我可以确认这适用于 MyPackage 示例和 one shown in the JuliaLang issue。感谢那些为这个答案做出贡献的人。

包设置不会传播给分布式工作人员

Package set-up not propagating to workers with Distributed

julia

include

loading

distributed