在scikit-learn的packages中指定say n_jobs=-1时,需要先导入joblib.Parallel吗?
When specifying say n_jobs=-1 in scikit-learn's packages, do we need to first import joblib.Parallel?
当使用 scikit-learn 的包可以选择 n_jobs
的值进行并行处理时,我们是否需要先导入 joblib.Parallel
或者 scikit-learn 包是否可以并行处理无需先导入 joblib.Parallel
.
即可处理
一些具有并行处理能力的 scikit-learn 包是:
sklearn.linear_model.LogisticRegression
xgboost.XGBRegressor
xgboost.XGBClassifier
等等
Q : "... do we need to first import joblib.Parallel
or will the scikit-learn package work with parallel processing without needing to first import joblib.Parallel
." ?
B 是正确的:scikit-learn 会工作,因为它经过适当的设计和实现,以便管理它自己的内部需求,以导入它有意依赖的任何包.这是一个专业的软件标准,适当照顾其自身的内部依赖性,不是吗?
奖励部分 indeed interested in :
那些确实想看到技巧的人 - 检查在其中一个包源代码级别上完成的继承方式,从下面报告的文件中显而易见:
(base) Tue Mar 17 12:00:34 a64FX:~$ grep -R "joblib.Parallel" /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py: parallel : joblib.Parallel (optional)
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py: Pre-initialized instance of joblib.Parallel.
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py: parallel : joblib.Parallel
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py: Pre-initialized instance of joblib.Parallel
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/__pycache__/online_lda.cpython-35.pyc matches
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py: print("Using %s as joblib.Parallel backend instead of %s "
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py: """Callback used by joblib.Parallel's multiprocessing backend.
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/_dask.py: joblib.Parallel will never access those results
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/_dask.py: # See 'joblib.Parallel.__call__' and 'joblib.Parallel.retrieve' for how
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/__pycache__/parallel.cpython-35.pyc matches
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/__pycache__/_dask.cpython-35.pyc matches
当使用 scikit-learn 的包可以选择 n_jobs
的值进行并行处理时,我们是否需要先导入 joblib.Parallel
或者 scikit-learn 包是否可以并行处理无需先导入 joblib.Parallel
.
一些具有并行处理能力的 scikit-learn 包是:
sklearn.linear_model.LogisticRegression
xgboost.XGBRegressor
xgboost.XGBClassifier
等等
Q : "... do we need to first import
joblib.Parallel
or will the scikit-learn package work with parallel processing without needing to first importjoblib.Parallel
." ?
B 是正确的:scikit-learn 会工作,因为它经过适当的设计和实现,以便管理它自己的内部需求,以导入它有意依赖的任何包.这是一个专业的软件标准,适当照顾其自身的内部依赖性,不是吗?
奖励部分 indeed interested in :
那些确实想看到技巧的人 - 检查在其中一个包源代码级别上完成的继承方式,从下面报告的文件中显而易见:
(base) Tue Mar 17 12:00:34 a64FX:~$ grep -R "joblib.Parallel" /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py: parallel : joblib.Parallel (optional)
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py: Pre-initialized instance of joblib.Parallel.
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py: parallel : joblib.Parallel
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py: Pre-initialized instance of joblib.Parallel
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/__pycache__/online_lda.cpython-35.pyc matches
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py: print("Using %s as joblib.Parallel backend instead of %s "
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py: """Callback used by joblib.Parallel's multiprocessing backend.
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/_dask.py: joblib.Parallel will never access those results
/home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/_dask.py: # See 'joblib.Parallel.__call__' and 'joblib.Parallel.retrieve' for how
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/__pycache__/parallel.cpython-35.pyc matches
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/__pycache__/_dask.cpython-35.pyc matches