srun 在循环中使用：srun：作业步骤已中止：等待作业步骤完成最多 32 秒

Question

我通过 srun 获得了一个 .sh 文件到运行因为我想看到脚本的动态打印输出。但是通过运行ning srun job_spinup.sh southfr_exp 1 & 我总是在 2 个主循环后出现错误（由于时间限制错误而超时）...这是 .sh 文件中的主要代码。顺便说一下，我想要运行一个 12 个月的模型并将其循环 20 次（所谓的旋转 20 次）。但是错误发生在第二次循环（旋转）的 11 月...... 这是 job_spinup.sh:

中的代码

#!/bin/bash
#SBATCH -J spinup
#SBATCH -p knl_cache
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 10:00:00
#SBATCH -o spinup.log
#SBATCH -e spinup.log
#=========================================================================
# USAGE
#   nohup ./job_spinup DOM[:EXP] nodes[:tasks_per_node:tasks_for_trip N START_ID:START_MM] &
#
# by default: EXP=spinup, N=20, START_ID=0, START_MM=1
#=========================================================================
#set -x
#
if [ $# -lt 2 ]; then
  echo "Usage: [=10=] DOM[:EXP:VERSION] nodes[:tasks_per_node:tasks_for_trip N START_ID:START_MM]"
  echo "DOM            = the name of a domain"
  echo "EXP            = the name of an experiment"
  echo "N              = the number of runnings"
  echo "START_ID       = start id of a running"
  echo "START_MM       = start month of a running"
  exit
fi

DOM=`echo  | awk '{split(, f, ":"); print f[1]}'`
EXP=`echo  | awk '{split(, f, ":"); print f[2]}'`
EXP=${EXP:-spinup}
VERSION=`echo  | awk '{split(, f, ":"); print f[3]}'`
VERSION=${VERSION:--X0}
num_nodes=`echo  | awk '{split(, f, ":"); print f[1]}'`
tasks_per_node=`echo  | awk '{split(, f, ":"); print f[2]}'`
tasks_per_node=${tasks_per_node:-40}
tasks_for_trip=`echo  | awk '{split(, f, ":"); print f[3]}'`
tasks_for_trip=${tasks_for_trip:-1}
SPINUP_N=${3:-20}
START_ID=`echo  | awk '{split(, f, ":"); print f[1]}'`
START_ID=${START_ID:-0}
START_MM=`echo  | awk '{split(, f, ":"); print f[2]}'`
START_MM=${START_MM:-1}

# source ~/anaconda3/etc/profile.d/conda.sh
source $(conda info --base)/etc/profile.d/conda.sh
conda activate myenv
echo "***************************************"
echo " CONDA ENV ACTIVATED FOR NCO COMMAND"
echo "***************************************"
echo $SPINUP_N
#
# check if TRIP is used
LTRIP=`grep "LOASIS *= *T" OPTIONS/OPTIONS.nam | wc -l`
#
ulimit -s unlimited
ulimit -n 500000
ulimit -u 64000
unset I_MPI_PMI_LIBRARY
export OMP_NUM_THREADS=1
export DR_HOOK=0
export DR_HOOK_OPT=prof

...

YYYY=${YYYYMMDDHH::4}
MM=${YYYYMMDDHH:4:2}
j=$START_ID
while [ $j -lt $SPINUP_N ] ; do

  echo " "
  echo "------------------"
  echo "SPINUP : $j / $SPINUP_N"

  while [ $MM -le 12 ] ; do
    if [ $LTRIP -eq 1 ]; then
      mpirun -np $((SLURM_NTASKS - tasks_for_trip)) offline.exe : -np $tasks_for_trip trip.exe &> offline
    else
      #echo ${SLURM_NTASKS}
      #mpirun -np ${SLURM_NTASKS} offline.exe &> offline
      #srun -n 1 offline.exe &> offline
      offline.exe &> offline
    fi
....

# Change dates to start again
    if [ $MM -eq 12 ]; then
      ncap2 -O -s "'DTCUR-YEAR'=$YYYY;'DTCUR-MONTH'=1;'DTCUR-DAY'=1;'DTCUR-TIME'=0" PREP.nc PREP.nc
      [ $LTRIP -eq 1 ] && ncap2 -O -s "date(:)={$YYYY,1,1,0}" TRIP_PREP.nc TRIP_PREP.nc
    fi

...


  done

  echo '------------------'
  echo ' '

  MM=01
  j=$(( j+1 ))

done
...
# end simulation
date >> date_$EXP
echo "***************************************"
echo "   SPINUP ENDS CORRECTLY"
echo "***************************************"

conda deactivate
echo "***************************************"
echo "   CONDA ENV DEACTIVATED"
echo "***************************************"

输出是这样的：

(base) [xushan@int2 southfr_exp]$ srun job_spinup.sh southfr_exp 1 &
[1] 11570
(base) [xushan@int2 southfr_exp]$ srun: job 8860513 queued and waiting for resources
srun: job 8860513 has been allocated resources
***************************************
 CONDA ENV ACTIVATED FOR NCO COMMAND
***************************************
20
./job_spinup.sh: line 62: ulimit: open files: cannot modify limit: Operation not permitted
***************************************
   READY TO START SPINUP on tcn991.bullx
     spinup 20 0:1
***************************************
 
------------------
SPINUP : 0 / 20
    199601
1
    199602
1
    199603
1
    199604
1
    199605
1
    199606
1
    199607
1
    199608
1
    199609
1
    199610
1
    199611
1
    199612
1
------------------
 
 
------------------
SPINUP : 1 / 20
    199601
1
    199602
1
    199603
1
    199604
1
    199605
1
    199606
1
    199607
1
    199608
1
    199609
1
    199610
1
srun: Force Terminated job 8860513
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 8860513.0 ON tcn991 CANCELLED AT 2020-09-07T12:51:24 DUE TO TIME LIMIT ***
srun: error: tcn991: task 0: Terminated
srun: Terminating job step 8860513.0

有没有人可以帮助我？多谢！我是slurm的初学者.....是不是因为我激活了conda环境？通过 squeue，我可以看到队列只持续了 5 分钟……不知道为什么……是因为 offline.exe?

Answer 1

srun 不像 sbatch 那样阅读作业脚本。这意味着您的所有 #SBATCH 选项都将被忽略，包括您为作业设置的时间限制。因此，您的作业会转到具有默认时间限制的默认分区，这似乎只够两个循环的时间。

有多种解决方法：

使用 sbatch 并查看输出文件 (tail -f spinup.log)
使用 sbatch 并使用 sattach
将 #SBATCH 选项作为参数添加到 srun

srun 在循环中使用：srun：作业步骤已中止：等待作业步骤完成最多 32 秒

srun used in a loop: srun: Job step aborted: Waiting up to 32 seconds for job step to finish

linux

mpi

nco

slurm

sbatch