Nvidia Visual profiler Error: Cannot create profiling file

Question

我是 nvprof 的新用户，当我尝试使用 nvprof 分析我的代码时遇到错误。我在我的代码中添加了一些 OpenACC 指令。我用的cuda工具包是Cuda8.0。我的代码是用 fortran90+OpenMPI 编写的。我使用 16 个内核进行并行计算。这是我用来提交代码的脚本：

#!/bin/bash -l
#
#SBATCH --nodes=1
#SBATCH --ntasks=16
#SBATCH --ntasks-per-node=24
#SBATCH --ntasks-per-core=2
#SBATCH --cpus-per-task=1
#SBATCH --constraint=gpu
#SBATCH --time=24:00:00
#SBATCH --account=s807

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export CRAY_CUDA_MPS=1
export TMPDIR=/scratch/snx3000/guow/Incompact3d_GPU/test1
export PGI_ACC_TIME=1

set -ex

# set some parameters
OUT=log.txt
#WMIN=1400
NP=16


# tasks: $SLURM_NTASKS 
# tasks-per-node: $SLURM_NTASKS_PER_NODE 
# cpus-per-task: $SLURM_CPUS_PER_TASK
srun nvprof -o nvprof.out ./incompact3d $WMIN >> $OUT

作业在运行几秒钟后终止，我收到以下错误消息：

+ OUT=log.txt
+ NP=16
+ srun nvprof -o nvprof.out ./incompact3d
==20144== NVPROF is profiling process 20144, command: ./incompact3d
==20148== NVPROF is profiling process 20148, command: ./incompact3d
==20140== NVPROF is profiling process 20140, command: ./incompact3d
==20133== NVPROF is profiling process 20133, command: ./incompact3d
==20134== NVPROF is profiling process 20134, command: ./incompact3d
==20150== NVPROF is profiling process 20150, command: ./incompact3d
==20146== NVPROF is profiling process 20146, command: ./incompact3d
==20128== NVPROF is profiling process 20128, command: ./incompact3d
==20154== NVPROF is profiling process 20154, command: ./incompact3d
==20156== NVPROF is profiling process 20156, command: ./incompact3d
==20152== NVPROF is profiling process 20152, command: ./incompact3d
==20136== NVPROF is profiling process 20136, command: ./incompact3d
==20130== NVPROF is profiling process 20130, command: ./incompact3d
==20158== NVPROF is profiling process 20158, command: ./incompact3d
==20138== NVPROF is profiling process 20138, command: ./incompact3d
==20142== NVPROF is profiling process 20142, command: ./incompact3d
==20146== Error: Cannot create profiling file: /scratch/snx3000/guow/Incompact3d_GPU/test2_OpenAcc/nvprof.out
==20142== Error: Cannot create profiling file: /scratch/snx3000/guow/Incompact3d_GPU/test2_OpenAcc/nvprof.out
==20138== Error: Cannot create profiling file: /scratch/snx3000/guow/Incompact3d_GPU/test2_OpenAcc/nvprof.out
==20133== Error: Cannot create profiling file: /scratch/snx3000/guow/Incompact3d_GPU/test2_OpenAcc/nvprof.out
==20134== Error: Cannot create profiling file: /scratch/snx3000/guow/Incompact3d_GPU/test2_OpenAcc/nvprof.out
==20136== Error: Cannot create profiling file: /scratch/snx3000/guow/Incompact3d_GPU/test2_OpenAcc/nvprof.out
==20156== Error: Cannot create profiling file: /scratch/snx3000/guow/Incompact3d_GPU/test2_OpenAcc/nvprof.out
srun: First task exited 30s ago
srun: tasks 0,4,6-7,9-13: running
srun: tasks 1-3,5,8,14-15: exited
srun: Terminating job step 3892463.0
slurmstepd: error: *** STEP 3892463.0 ON nid04439 CANCELLED AT 2017-10-12T15:04:00 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: nid04439: tasks 0,4,6-7,9-13: Killed
srun: Terminating job step 3892463.0

生成了一个nvprof.out文件，但是当我使用命令"nvvp nvprof.out"检查它时没有显示任何内容。你以前遇到过这种情况吗？任何建议将不胜感激！

Answer 1

尝试使用命令：

srun nvprof -o nvprof.%p.out ./incompact3d $WMIN >> $OUT

"%p" 将填充每个 MPI Rank 的进程 ID，因此每个 Rank 会将其配置文件输出到不同的文件。否则，所有队伍都在尝试使用同一个文件，这可能会导致问题。

Nvidia Visual profiler Error: Cannot create profiling file

Nvidia Visual profiler Error: Cannot create profiling file

profiling

cuda

pgi

openacc

nvprof