正确使用 cudaFortran cuSolver 函数
Proper use of cudaFortran cuSolver functions
我目前正在努力将一些 Fortran 代码迁移到 cudaFortran。具体来说，该任务涉及大规模矩阵的频谱分析，以便对角化它们。这是我到目前为止编写的代码
program main
!Trials for usage of cusovlerDn<t>syevd for spectral analysis of a symmetric matrix, see http://docs.nvidia.com/cuda/cusolver/index.html#syevd-example1 for the example used as a base
!Compilation example: 'pgf90 Main.cuf -lcusolver -Mcuda=cuda8.0'
use cudafor !has to go first
use cusolverDn
implicit none
integer :: info
integer,parameter :: q2 = SELECTED_REAL_KIND(15,305)
real(q2), device, dimension(3,3) :: A_d
real(q2), dimension(3,3) :: A
real(q2), device, dimension(3) :: W_d
real(q2), dimension(3) :: W
integer :: stat, lwork, m, lda
real(q2), device, allocatable :: work_d(:)
integer, device :: devInfo
type(cusolverDnHandle) :: h
stat=cusolverDnCreate(h)
W_d=(/0,0,0/)
print *, stat
m=3
lda = m
A_d(1,1:3)=(/4,1,2/)
A_d(2,1:3)=(/1,-1,1/)
A_d(3,1:3)=(/2,1,3/) !eigenvalues are 5.84947, 1.44865, -1.29812
! A_d(1,1:3)=(/1,0,0/)
! A_d(2,1:3)=(/0,1,0/)
! A_d(3,1:3)=(/0,0,1/)
stat=cusolverDnDsyevd_bufferSize(h, CUSOLVER_EIG_MODE_NOVECTOR, CUBLAS_FILL_MODE_UPPER, m, A_d, lda, W_d, lwork)
print *, stat
allocate(work_d(lwork))
stat=cusolverDnDsyevd(h, CUSOLVER_EIG_MODE_NOVECTOR, CUBLAS_FILL_MODE_UPPER, m, A_d, lda, W_d, work_d, lwork, devInfo)
print *, stat !returns 6 as if there was an error
info=devInfo
print *, info !devInfo returns 0, as if the operation was successful
stat=cudaDeviceSynchronize()
print *, stat
W=W_d
print *, W
A=A_d
print *, A
deallocate(work_d)
stat=cusolverDnDestroy(h)
print *, stat
end program main
编译和mem-check输出如下：
olafur@olafur-X556UQK:~/Skyrmions2017/Project$ pgf90 Main.cuf -lcusolver -Mcuda=cuda8.0
olafur@olafur-X556UQK:~/Skyrmions2017/Project$ cuda-memcheck ./a.out
========= CUDA-MEMCHECK
0
0
========= Program hit cudaErrorInvalidDeviceFunction (error 8) due to "invalid device function" on CUDA API call to cudaLaunch.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x5b906e]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e0857]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e0270]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e3df3]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e1720]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e0157]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 (cusolverDnDsytrd + 0x37) [0x2e3f17]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2ea607]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2eb744]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 (cusolverDnDsyevd + 0x27) [0x2ea157]
========= Host Frame:./a.out [0x1b2d]
========= Host Frame:./a.out [0x1514]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
========= Host Frame:./a.out [0x13f9]
=========
6
========= Program hit cudaErrorInvalidDeviceFunction (error 8) due to "invalid device function" on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x5b6793]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e1727]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e0157]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 (cusolverDnDsytrd + 0x37) [0x2e3f17]
0
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2ea607]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2eb744]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 (cusolverDnDsyevd + 0x27) [0x2ea157]
========= Host Frame:./a.out [0x1b2d]
0
========= Host Frame:./a.out [0x1514]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
========= Host Frame:./a.out [0x13f9]
=========
0.000000000000000 0.000000000000000 0.000000000000000
4.000000000000000 1.000000000000000 2.000000000000000
1.000000000000000 -1.000000000000000 1.000000000000000
2.000000000000000 1.000000000000000 3.000000000000000
0
========= ERROR SUMMARY: 2 errors
看起来我没有正确调用 cusolverDnDsyevd 函数，很可能我没有使用正确类型的变量。但是由于我在编程方面是半文盲而且我必须遵循的唯一示例是用 C 编写的（使用那些花哨的 void** 东西）我不知道什么是正确的。
编辑：deviceQuery
的完整输出
olafur@olafur-X556UQK:~/NVIDIA_CUDA-8.0_Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce 940MX"
CUDA Driver Version / Runtime Version 8.0 / 8.0
CUDA Capability Major/Minor version number: 5.0
Total amount of global memory: 2002 MBytes (2099642368 bytes)
( 3) Multiprocessors, (128) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 1242 MHz (1.24 GHz)
Memory Clock rate: 900 Mhz
Memory Bus Width: 64-bit
L2 Cache Size: 1048576 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce 940MX
Result = PASS
由于代码在我处理的另一个系统上运行良好，问题确实是运行时环境问题，正如 Robert Crovella 所建议的那样
故事的寓意：始终至少尝试 2 个系统。
我目前正在努力将一些 Fortran 代码迁移到 cudaFortran。具体来说，该任务涉及大规模矩阵的频谱分析，以便对角化它们。这是我到目前为止编写的代码
program main
!Trials for usage of cusovlerDn<t>syevd for spectral analysis of a symmetric matrix, see http://docs.nvidia.com/cuda/cusolver/index.html#syevd-example1 for the example used as a base
!Compilation example: 'pgf90 Main.cuf -lcusolver -Mcuda=cuda8.0'
use cudafor !has to go first
use cusolverDn
implicit none
integer :: info
integer,parameter :: q2 = SELECTED_REAL_KIND(15,305)
real(q2), device, dimension(3,3) :: A_d
real(q2), dimension(3,3) :: A
real(q2), device, dimension(3) :: W_d
real(q2), dimension(3) :: W
integer :: stat, lwork, m, lda
real(q2), device, allocatable :: work_d(:)
integer, device :: devInfo
type(cusolverDnHandle) :: h
stat=cusolverDnCreate(h)
W_d=(/0,0,0/)
print *, stat
m=3
lda = m
A_d(1,1:3)=(/4,1,2/)
A_d(2,1:3)=(/1,-1,1/)
A_d(3,1:3)=(/2,1,3/) !eigenvalues are 5.84947, 1.44865, -1.29812
! A_d(1,1:3)=(/1,0,0/)
! A_d(2,1:3)=(/0,1,0/)
! A_d(3,1:3)=(/0,0,1/)
stat=cusolverDnDsyevd_bufferSize(h, CUSOLVER_EIG_MODE_NOVECTOR, CUBLAS_FILL_MODE_UPPER, m, A_d, lda, W_d, lwork)
print *, stat
allocate(work_d(lwork))
stat=cusolverDnDsyevd(h, CUSOLVER_EIG_MODE_NOVECTOR, CUBLAS_FILL_MODE_UPPER, m, A_d, lda, W_d, work_d, lwork, devInfo)
print *, stat !returns 6 as if there was an error
info=devInfo
print *, info !devInfo returns 0, as if the operation was successful
stat=cudaDeviceSynchronize()
print *, stat
W=W_d
print *, W
A=A_d
print *, A
deallocate(work_d)
stat=cusolverDnDestroy(h)
print *, stat
end program main
编译和mem-check输出如下：
olafur@olafur-X556UQK:~/Skyrmions2017/Project$ pgf90 Main.cuf -lcusolver -Mcuda=cuda8.0
olafur@olafur-X556UQK:~/Skyrmions2017/Project$ cuda-memcheck ./a.out
========= CUDA-MEMCHECK
0
0
========= Program hit cudaErrorInvalidDeviceFunction (error 8) due to "invalid device function" on CUDA API call to cudaLaunch.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x5b906e]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e0857]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e0270]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e3df3]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e1720]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e0157]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 (cusolverDnDsytrd + 0x37) [0x2e3f17]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2ea607]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2eb744]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 (cusolverDnDsyevd + 0x27) [0x2ea157]
========= Host Frame:./a.out [0x1b2d]
========= Host Frame:./a.out [0x1514]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
========= Host Frame:./a.out [0x13f9]
=========
6
========= Program hit cudaErrorInvalidDeviceFunction (error 8) due to "invalid device function" on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x5b6793]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e1727]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2e0157]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 (cusolverDnDsytrd + 0x37) [0x2e3f17]
0
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2ea607]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 [0x2eb744]
========= Host Frame:/opt/pgi/linux86-64/2017/cuda/8.0/lib64/libcusolver.so.8.0 (cusolverDnDsyevd + 0x27) [0x2ea157]
========= Host Frame:./a.out [0x1b2d]
0
========= Host Frame:./a.out [0x1514]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
========= Host Frame:./a.out [0x13f9]
=========
0.000000000000000 0.000000000000000 0.000000000000000
4.000000000000000 1.000000000000000 2.000000000000000
1.000000000000000 -1.000000000000000 1.000000000000000
2.000000000000000 1.000000000000000 3.000000000000000
0
========= ERROR SUMMARY: 2 errors
看起来我没有正确调用 cusolverDnDsyevd 函数，很可能我没有使用正确类型的变量。但是由于我在编程方面是半文盲而且我必须遵循的唯一示例是用 C 编写的（使用那些花哨的 void** 东西）我不知道什么是正确的。
编辑：deviceQuery
olafur@olafur-X556UQK:~/NVIDIA_CUDA-8.0_Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce 940MX"
CUDA Driver Version / Runtime Version 8.0 / 8.0
CUDA Capability Major/Minor version number: 5.0
Total amount of global memory: 2002 MBytes (2099642368 bytes)
( 3) Multiprocessors, (128) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 1242 MHz (1.24 GHz)
Memory Clock rate: 900 Mhz
Memory Bus Width: 64-bit
L2 Cache Size: 1048576 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce 940MX
Result = PASS
由于代码在我处理的另一个系统上运行良好，问题确实是运行时环境问题，正如 Robert Crovella 所建议的那样
故事的寓意：始终至少尝试 2 个系统。