Matlab parfor 使用的内核数少于分配的内核数

Question

我正在运行在远程 cluster 的单个节点上执行并行 Matlab 作业。集群的每个节点都有 2 个处理器，每个处理器有 24 个内核，每个节点总共有 48 个内核。该作业包含一些顺序代码，后跟一个 parfor 循环。我运行它使用 slurm bash 脚本。

bash脚本test.sh是：

#!/bin/bash
#
########## Begin Slurm header ##########
#
# Give job a reasonable name
#SBATCH -J test_1
#
# Request number of nodes and CPU cores per node for job
#SBATCH --nodes=1
# Request number of tasks/process per nodes
# (determines number of workers in processed based parpool)
#SBATCH --tasks-per-node=48
# Estimated wallclock time for job
#SBATCH -t 1-00
#
# Send mail when job begins, aborts and ends
#SBATCH --mail-type=ALL
#
########### End Slurm header ##########

echo "Submit Directory:                     $SLURM_SUBMIT_DIR"
echo "Working Directory:                    $PWD"
echo "Running on host                       $HOSTNAME"
echo "Job id:                               $SLURM_JOB_ID"
echo "Job name:                             $SLURM_JOB_NAME"
echo "Number of nodes allocated to job:     $SLURM_JOB_NUM_NODES"
echo "Number of cores allocated to job:     $SLURM_NPROCS"
echo "Number of requested tasks per node:   $SLURM_NTASKS_PER_NODE"

# Load module
module load math/matlab/R2020a

#   Create a local working directory on scratch
mkdir -p $SCRATCH/$SLURM_JOB_ID

# Start a Matlab program
matlab -nodisplay -batch test_1 > test_1.out 2>&1

# Cleanup local working directory
rm -rf $SCRATCH/$SLURM_JOB_ID

exit

Matlab 脚本是

% Create parallel pool

pc = parcluster('local');

pc.JobStorageLocation = strcat(getenv('SCRATCH'),'/',getenv('SLURM_JOB_ID'));
 
num_workers = str2double(getenv('SLURM_NPROCS'));
parpool(pc,num_workers);

% Body of the script

% Choose deterministic parameters

free_points = 845000;
pulse_points = 1300000;
dt = 2e-11;

num_freqs = 200;
freqs = linspace(-1,1,200);

rhoi = rand(72);
rhoi = rhoi + rhoi';
rhoi = rhoi/trace(rhoi);

% Iterate over random parameters

num_pars = 5;
res = zeros(num_pars,num_freqs);
for n=1:num_pars

    disp('=====');
    disp(['N = ',num2str(n)]);
    disp('=====');

    timer = tic;

    % Random parameters

    H = rand(size(rhoi));
    H = (H + H')/2;

    L1 = rand(size(rhoi));
    L2 = rand(size(rhoi));
    L3 = rand(size(rhoi));
    L4 = rand(size(rhoi));
    L5 = rand(size(rhoi));

    % Equation to solve

    ME = @(rhot, t, w)  -1i*w*(H*rhot - rhot*H) + (L1*rhot*L1' - (1/2)*rhot*L1'*L1 - (1/2)*L1'*L1*rhot) ...
                                                + (L2*rhot*L2' - (1/2)*rhot*L2'*L2 - (1/2)*L2'*L2*rhot) ...
                                                + (L3*rhot*L3' - (1/2)*rhot*L3'*L3 - (1/2)*L3'*L3*rhot) ...
                                                + (L4*rhot*L4' - (1/2)*rhot*L4'*L4 - (1/2)*L4'*L4*rhot) ...
                                                + (L5*rhot*L5' - (1/2)*rhot*L5'*L5 - (1/2)*L5'*L5*rhot);

    % Solve equation

    % IF I CHANGE TO 'for j = 1:1', ALL WORKERS ARE USED!!! MEMORY?
    for j = 1:free_points
       rhoi =  RK4(@(rho, t) ME(rho, t, 0), rhoi, j, dt);
    end

    t = toc(timer);
    disp(['Mid duration ',num2str(t),'s']);

    parfor k=1:num_freqs
        w = freqs(k);
        
        rhop = rhoi;
        
        for j=1:pulse_points
            rhop = RK4(@(rho, t) ME(rho, t, w), rhop, j, dt);
        end
        
        for j=1:free_points
            rhop = RK4(@(rho, t) ME(rho, t, 0), rhop, j, dt);
        end

        occ(k) = rhop(1,1);
    end

    % Store result

    res(n,:) = occ;

end

save('res','res');

% Delete the parallel pool

delete(gcp('nocreate'));


% Local functions

function [rho] = RK4(F, rho, k, h)

k1 = F(rho, k*h);
k2 = F(rho+h*k1/2, (k+1/2)*h);
k3 = F(rho+h*k2/2, (k+1/2)*h);
k4 = F(rho+h*k3, (k+1)*h);

rho = rho+(1/6)*h*(k1+2*k2+2*k3+k4);

end

slurm输出是

#
# SOME PERSONAL INFO HERE...
#
Number of nodes allocated to job:     1
Number of cores allocated to job:     48
Number of requested tasks per node:   48
 IMPORTANT: The MATLAB Academic site license is available to employees and
 enrolled students of the the universities of (CENSORED).
 The license is available for teaching or research only.
 Commercial applications are not permitted.

Matlab 输出为

Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 48).
=====
N = 1
=====
Mid duration 3608.9535s
Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 12).

#
# REST OF OUTPUT HERE...
#

您看到当 Matlab 脚本启动时，创建了一个包含 48 个工人的池。但是当 parfor 循环最终开始时，parpool 重新启动并且工人数量降级为 12。

我注意到只有当循环的大小足够大时才会发生这种情况，即使是非parfor循环。例如，如果我将第一个 for 循环的大小更改为 1，则 parpool 不会重新启动。所以我认为这可能与内存使用有关...？

知道发生了什么以及如何让 Matlab 使用分配的所有 48 个内核吗？

编辑：我尝试过的另一件事是删除 parpool 命令并将 parfor 循环中的集群指定为 parfor (k=1:num_freqs,pc)。当我这样做时，无论我的循环大小如何，Matlab 都会使用四分之一的工人。我会尝试直接联系管理员...

Answer 1

我敢打赌您的并行池在 parfor 循环之间超时。然后它会自动创建，大小为 12，因为这是“并行池中的首选工作人员数量”(doc) 的默认首选项。（就个人而言，我不太在意这种偏好，总是将值设置为 99999 并让其他东西控制池的大小，但在你的情况下，如果你的 SLURM 工作人员不共享一个，你可能无法做到这一点与您的客户一起使用 MATLAB 首选项目录 (prefdir)。

我建议您使用 Inf 的 IdleTimeout 创建大小为 48 的池，如下所示：

num_workers = str2double(getenv('SLURM_NPROCS'));
parpool(pc,num_workers,'IdleTimeout',Inf);

Matlab parfor 使用的内核数少于分配的内核数

Matlab parfor uses fewer cores than the allocated number of cores

parallel-processing

bash

matlab

parfor

slurm