MPI_alltoallw 工作和 MPI_Ialltoallw 失败

Question

我正在尝试在大型代码中实现非阻塞通信。但是，对于这种情况，代码往往会失败。我已经重现了下面的错误。当运行在一个 CPU 上时，下面的代码在 switch 设置为 false 时有效，但在 switch 设置为 true 时失败。

program main
      
  use mpi

  implicit none

  logical :: switch
  integer, parameter :: maxSize=128
  integer scounts(maxSize), sdispls(maxSize)
  integer rcounts(maxSize), rdispls(maxSize)
  integer :: types(maxSize)
  double precision sbuf(maxSize), rbuf(maxSize)
  integer comm, size, rank, req
  integer ierr
  integer ii

  call MPI_INIT(ierr)
  comm = MPI_COMM_WORLD
  call MPI_Comm_size(comm, size, ierr)
  call MPI_Comm_rank(comm, rank, ierr)

  switch = .true.

  ! Init
  sbuf(:) = rank
  scounts(:) = 0
  rcounts(:) = 0
  sdispls(:) = 0
  rdispls(:) = 0
  types(:) = MPI_INTEGER
  if (switch) then
    ! Send one time N double precision
    scounts(1)  = 1
    rcounts(1)  = 1
    sdispls(1)  = 0
    rdispls(1)  = 0
    call MPI_Type_create_subarray(1, (/maxSize/), &
                                     (/maxSize/), &
                                     (/0/), &
                                     MPI_ORDER_FORTRAN,MPI_DOUBLE_PRECISION, &
                                     types(1),ierr)
    call MPI_Type_commit(types(1),ierr)
  else
    ! Send N times one double precision
    do ii = 1, maxSize
      scounts(ii)  = 1
      rcounts(ii)  = 1
      sdispls(ii)  = ii-1
      rdispls(ii)  = ii-1
      types(ii)    = MPI_DOUBLE_PRECISION
    enddo
  endif

  call MPI_Ibarrier(comm, req, ierr)
  call MPI_Wait(req, MPI_STATUS_IGNORE, ierr)

  if (switch) then
    call MPI_Ialltoallw(sbuf, scounts, sdispls, types, &
                        rbuf, rcounts, rdispls, types, &
                        comm, req, ierr)
    call MPI_Wait(req, MPI_STATUS_IGNORE, ierr)
    call MPI_TYPE_FREE(types(1), ierr)
  else
    call MPI_alltoallw(sbuf, scounts, sdispls, types, &
                       rbuf, rcounts, rdispls, types, &
                       comm, ierr)
  endif

  call MPI_Finalize( ierr )

end program main

使用调试标志和运行 mpirun -np 1 valgrind --vgdb=yes --vgdb-error=0 ./a.out 编译会导致 valgrind 和 gdb 出现以下错误：

valgrind :
==249074== Invalid read of size 8
==249074==    at 0x4EB0A6D: release_vecs_callback (coll_base_util.c:222)
==249074==    by 0x4EB100A: complete_vecs_callback (coll_base_util.c:245)
==249074==    by 0x74AD1CC: ompi_request_complete (request.h:441)
==249074==    by 0x74AE86D: ompi_coll_libnbc_progress (coll_libnbc_component.c:466)
==249074==    by 0x4FC0C39: opal_progress (opal_progress.c:231)
==249074==    by 0x4E04795: ompi_request_wait_completion (request.h:415)
==249074==    by 0x4E047EB: ompi_request_default_wait (req_wait.c:42)
==249074==    by 0x4E80AF7: PMPI_Wait (pwait.c:74)
==249074==    by 0x48A30D2: mpi_wait (pwait_f.c:76)
==249074==    by 0x10961A: MAIN__ (tmp.f90:61)
==249074==    by 0x1096C6: main (tmp.f90:7)
==249074==  Address 0x7758830 is 0 bytes inside a block of size 8 free'd
==249074==    at 0x483CA3F: free (vg_replace_malloc.c:540)
==249074==    by 0x4899CCC: PMPI_IALLTOALLW (pialltoallw_f.c:125)
==249074==    by 0x1095FC: MAIN__ (tmp.f90:61)
==249074==    by 0x1096C6: main (tmp.f90:7)
==249074==  Block was alloc'd at
==249074==    at 0x483B7F3: malloc (vg_replace_malloc.c:309)
==249074==    by 0x4899B4A: PMPI_IALLTOALLW (pialltoallw_f.c:90)
==249074==    by 0x1095FC: MAIN__ (tmp.f90:61)
==249074==    by 0x1096C6: main (tmp.f90:7)
gdb :
Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
0x0000000004eb0a6d in release_vecs_callback (request=0x7758af8) at ../../../../openmpi-4.1.0/ompi/mca/coll/base/coll_base_util.c:222
222             if (NULL != request->data.vecs.stypes[i]) {
(gdb) bt
#0  0x0000000004eb0a6d in release_vecs_callback (request=0x7758af8) at ../../../../openmpi-4.1.0/ompi/mca/coll/base/coll_base_util.c:222
#1  0x0000000004eb100b in complete_vecs_callback (req=0x7758af8) at ../../../../openmpi-4.1.0/ompi/mca/coll/base/coll_base_util.c:245
#2  0x00000000074ad1cd in ompi_request_complete (request=0x7758af8, with_signal=true) at ../../../../../openmpi-4.1.0/ompi/request/request.h:441
#3  0x00000000074ae86e in ompi_coll_libnbc_progress () at ../../../../../openmpi-4.1.0/ompi/mca/coll/libnbc/coll_libnbc_component.c:466
#4  0x0000000004fc0c3a in opal_progress () at ../../openmpi-4.1.0/opal/runtime/opal_progress.c:231
#5  0x0000000004e04796 in ompi_request_wait_completion (req=0x7758af8) at ../../openmpi-4.1.0/ompi/request/request.h:415
#6  0x0000000004e047ec in ompi_request_default_wait (req_ptr=0x1ffeffdbb8, status=0x1ffeffdbc0) at ../../openmpi-4.1.0/ompi/request/req_wait.c:42
#7  0x0000000004e80af8 in PMPI_Wait (request=0x1ffeffdbb8, status=0x1ffeffdbc0) at pwait.c:74
#8  0x00000000048a30d3 in ompi_wait_f (request=0x1ffeffe6cc, status=0x10c0a0 <mpi_fortran_status_ignore_>, ierr=0x1ffeffeee0) at pwait_f.c:76
#9  0x000000000010961b in MAIN__ () at tmp.f90:61

如有任何帮助，我们将不胜感激。 Ubuntu 20.04。 gfortran 9.3.0。 OpenMP 4.1.0。谢谢。

Answer 1

建议的程序目前在使用 Open MPI, see issue https://github.com/open-mpi/ompi/issues/8763. The current workaround is to use MPICH 时损坏。

编辑：该错误已在 OpenMPI 的主要分支中修复，应该在 5.0 及更高版本中修复。

MPI_alltoallw 工作和 MPI_Ialltoallw 失败

MPI_alltoallw working and MPI_Ialltoallw failing

fortran

mpi

gfortran

openmpi