包修改和分布式循环的问题

Question

所以我想做一些计算，因此想并行化一个循环。我正在使用包 Revise 来应用我在代码中所做的更改。如果我在更改某些内容后重新启动完整的 julia 会话，这会很好用。

我尝试了一些不同的选项并简化了我的问题，但我找不到解决方案。

我创建了以下模块：

module ParallelTester

using Distributed
export paralleltest

function paralleltest()
    @sync @distributed for i in 1:100
        println("Hi there!")
    end
end

end

当我开始我的 julia 会话时，我会执行以下操作：

using Distributed
addprocs(3)
push!(LOAD_PATH, pwd())
@everywhere using Revise, ParallelTester

当我运行函数 paralleltest() 结果按预期打印。

当我现在将输出从 Hi there! 更改为 Hi there again! 时，出现此错误：

ERROR: On worker 2:
UndefVarError: ##5#6 not defined
deserialize_datatype at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:1115
handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:771
deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:731
deserialize_datatype at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:1139
handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:771
deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:731
handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:778
deserialize_msg at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Serialization\src\Serialization.jl:731
#invokelatest#1 at .\essentials.jl:742 [inlined]
invokelatest at .\essentials.jl:741 [inlined]
message_handler_loop at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:160
process_tcp_streams at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:117
#105 at .\task.jl:259
#remotecall_fetch#149(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Distributed.RRID, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:379
remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:371
#remotecall_fetch#152(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Distributed.RRID, ::Vararg{Any,N} where N) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:406
call_on_owner at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:406 [inlined]
wait(::Future) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:500
sync_end(::Array{Any,1}) at .\task.jl:213
(::getfield(Distributed, Symbol("##167#169")){getfield(Tester, Symbol("##5#6")),UnitRange{Int64}})() at .\task.jl:245
Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:226
 [2] macro expansion at .\task.jl:245 [inlined]
 [3] parallelTest() at C:\Users\FelixMeyer\AppData\Local\Julia-1.1.1\Tester.jl:9
 [4] top-level scope at none:0

我错过了什么？

我在 Revise 包中找到了这个 docspage，它说匿名函数可能会出现这样的错误。

Answer 1

解决方法是将 paralleltest 的 "guts" 放在单独的函数中：

function paralleltest(rng)
    @sync @distributed for i in rng
        paralleltestcore()
    end
end

paralleltestcore() = println("Hi there!")

但如果能解决这个问题就好了。如果我们降低主进程的代码，然后将降低的代码发送到每个进程进行评估，可能有一种方法可以做到这一点。同时，我已将此案例添加到 https://github.com/timholy/Revise.jl/issues/147。

包修改和分布式循环的问题

Problem with package Revise and distributed loop

parallel-processing

julia

distributed