运行 在 R 中进行并行计算时如何在工作人员上设置 .libPaths（检查点）
How to set .libPaths (checkpoint) on workers when running parallel computation in R
我使用检查点包进行可重现的数据分析。有些计算需要很长时间才能计算，所以我想 运行 并行处理这些计算。
当 运行 并行但是检查点没有设置在工人身上，所以我收到一条错误消息 "there is no package called xy" （因为它没有安装在我的默认库目录中).
我如何确保每个工作人员都使用检查点文件夹中的包版本？我试图在 foreach 代码中设置 .libPaths 但这似乎不起作用。我也更愿意在全局范围内而不是在每个 foreach 调用中设置 checkpoint/libPaths 一次。
另一种选择是更改 .Rprofile 文件，但我不想这样做。
checkpoint::checkpoint("2018-06-01")
library(foreach)
library(doFuture)
library(future)
doFuture::registerDoFuture()
future::plan("multisession")
l <- .libPaths()
# Code to run in parallel does not make much sense of course but I wanted to keep it simple.
res <- foreach::foreach(
x = unique(iris$Species),
lib.path = l
) %dopar% {
.libPaths(lib.path)
stringr::str_c(x, "_")
}
Error in { : task 2 failed - "there is no package called 'stringr'"
此处 future 包的作者。
更新 2022-05-25：从 future 1.20.0 (2021-11-03) 开始，multisession parallel workers 自动从主 R 会话继承 R 库路径 (=.libPaths())。因此，不再需要以下解决方法。但是，其他未来的后端可能仍然需要它。
将主R进程的库路径作为全局变量传递libs并为每个使用.libPaths(libs)的worker设置它应该就足够了；
## Use CRAN checkpoint from 2018-07-24 to get future (>= 1.9.0) [1],
## otherwise the below stdout won't be relayed back to the master
## R process, but settings .libPaths() does also work in older
## versions of the future package.
## [1] https://cran.microsoft.com/snapshot/2018-07-24/web/packages/future
checkpoint::checkpoint("2018-07-24")
stopifnot(packageVersion("future") >= "1.9.0")
libs <- .libPaths()
print(libs)
### [1] "/home/hb/.checkpoint/2018-07-24/lib/x86_64-pc-linux-gnu/3.5.1"
### [2] "/home/hb/.checkpoint/R-3.5.1"
### [3] "/usr/lib/R/library"
library(foreach)
doFuture::registerDoFuture()
future::plan("multisession")
res <- foreach::foreach(x = unique(iris$Species)) %dopar% {
## Use the same library paths as the master R session
.libPaths(libs)
cat(sprintf("Library paths used by worker (PID %d):\n", Sys.getpid()))
cat(sprintf(" - %s\n", sQuote(.libPaths())))
stringr::str_c(x, "_")
}
### - ‘/home/hb/.checkpoint/2018-07-24/lib/x86_64-pc-linux-gnu/3.5.1’
### - ‘/home/hb/.checkpoint/R-3.5.1’
### - ‘/usr/lib/R/library’
### Library paths used by worker (PID 9394):
### - ‘/home/hb/.checkpoint/2018-07-24/lib/x86_64-pc-linux-gnu/3.5.1’
### - ‘/home/hb/.checkpoint/R-3.5.1’
### - ‘/usr/lib/R/library’
### Library paths used by worker (PID 9412):
### - ‘/home/hb/.checkpoint/2018-07-24/lib/x86_64-pc-linux-gnu/3.5.1’
### - ‘/home/hb/.checkpoint/R-3.5.1’
### - ‘/usr/lib/R/library’
str(res)
### List of 3
### $ : chr "setosa_"
### $ : chr "versicolor_"
### $ : chr "virginica_"
仅供参考，它在 make it easier to pass down the library path(s) to workers 的未来路线图上。
我的详细信息：
> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] foreach_1.4.4
loaded via a namespace (and not attached):
[1] drat_0.1.4 compiler_3.5.1 BiocManager_1.30.2 parallel_3.5.1 tools_3.5.1 listenv_0.7.0 doFuture_0.6.0
[8] codetools_0.2-15 iterators_1.0.10 digest_0.6.15 globals_0.12.1 checkpoint_0.4.5 future_1.9.0
我使用检查点包进行可重现的数据分析。有些计算需要很长时间才能计算，所以我想 运行 并行处理这些计算。 当 运行 并行但是检查点没有设置在工人身上，所以我收到一条错误消息 "there is no package called xy" （因为它没有安装在我的默认库目录中).
我如何确保每个工作人员都使用检查点文件夹中的包版本？我试图在 foreach 代码中设置 .libPaths 但这似乎不起作用。我也更愿意在全局范围内而不是在每个 foreach 调用中设置 checkpoint/libPaths 一次。
另一种选择是更改 .Rprofile 文件，但我不想这样做。
checkpoint::checkpoint("2018-06-01")
library(foreach)
library(doFuture)
library(future)
doFuture::registerDoFuture()
future::plan("multisession")
l <- .libPaths()
# Code to run in parallel does not make much sense of course but I wanted to keep it simple.
res <- foreach::foreach(
x = unique(iris$Species),
lib.path = l
) %dopar% {
.libPaths(lib.path)
stringr::str_c(x, "_")
}
Error in { : task 2 failed - "there is no package called 'stringr'"
此处 future 包的作者。
更新 2022-05-25：从 future 1.20.0 (2021-11-03) 开始，multisession parallel workers 自动从主 R 会话继承 R 库路径 (=.libPaths())。因此，不再需要以下解决方法。但是，其他未来的后端可能仍然需要它。
将主R进程的库路径作为全局变量传递libs并为每个使用.libPaths(libs)的worker设置它应该就足够了；
## Use CRAN checkpoint from 2018-07-24 to get future (>= 1.9.0) [1],
## otherwise the below stdout won't be relayed back to the master
## R process, but settings .libPaths() does also work in older
## versions of the future package.
## [1] https://cran.microsoft.com/snapshot/2018-07-24/web/packages/future
checkpoint::checkpoint("2018-07-24")
stopifnot(packageVersion("future") >= "1.9.0")
libs <- .libPaths()
print(libs)
### [1] "/home/hb/.checkpoint/2018-07-24/lib/x86_64-pc-linux-gnu/3.5.1"
### [2] "/home/hb/.checkpoint/R-3.5.1"
### [3] "/usr/lib/R/library"
library(foreach)
doFuture::registerDoFuture()
future::plan("multisession")
res <- foreach::foreach(x = unique(iris$Species)) %dopar% {
## Use the same library paths as the master R session
.libPaths(libs)
cat(sprintf("Library paths used by worker (PID %d):\n", Sys.getpid()))
cat(sprintf(" - %s\n", sQuote(.libPaths())))
stringr::str_c(x, "_")
}
### - ‘/home/hb/.checkpoint/2018-07-24/lib/x86_64-pc-linux-gnu/3.5.1’
### - ‘/home/hb/.checkpoint/R-3.5.1’
### - ‘/usr/lib/R/library’
### Library paths used by worker (PID 9394):
### - ‘/home/hb/.checkpoint/2018-07-24/lib/x86_64-pc-linux-gnu/3.5.1’
### - ‘/home/hb/.checkpoint/R-3.5.1’
### - ‘/usr/lib/R/library’
### Library paths used by worker (PID 9412):
### - ‘/home/hb/.checkpoint/2018-07-24/lib/x86_64-pc-linux-gnu/3.5.1’
### - ‘/home/hb/.checkpoint/R-3.5.1’
### - ‘/usr/lib/R/library’
str(res)
### List of 3
### $ : chr "setosa_"
### $ : chr "versicolor_"
### $ : chr "virginica_"
仅供参考，它在 make it easier to pass down the library path(s) to workers 的未来路线图上。
我的详细信息：
> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] foreach_1.4.4
loaded via a namespace (and not attached):
[1] drat_0.1.4 compiler_3.5.1 BiocManager_1.30.2 parallel_3.5.1 tools_3.5.1 listenv_0.7.0 doFuture_0.6.0
[8] codetools_0.2-15 iterators_1.0.10 digest_0.6.15 globals_0.12.1 checkpoint_0.4.5 future_1.9.0