"CUDA_ERROR_INVALID_DEVICE: invalid device ordinal" 启动 TensorFlow 会话时
"CUDA_ERROR_INVALID_DEVICE: invalid device ordinal" when starting a TensorFlow session
启动 TensorFlow 会话时，未检测到 GPU (CUDA_ERROR_INVALID_DEVICE: invalid device ordinal)：
$ CUDA_VISIBLE_DEVICES='0' python3 -c 'import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))'
2019-07-18 09:36:55.661519: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-18 09:36:55.684438: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3312000000 Hz
2019-07-18 09:36:55.684721: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x41adbb0 executing computations on platform Host. Devices:
2019-07-18 09:36:55.684750: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2019-07-18 09:36:55.686513: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-18 09:36:55.696958: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
2019-07-18 09:36:55.697001: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: tobias-Z170-HD3P
2019-07-18 09:36:55.697006: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: tobias-Z170-HD3P
2019-07-18 09:36:55.697084: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 410.73.0
2019-07-18 09:36:55.697108: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 410.73.0
2019-07-18 09:36:55.697113: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 410.73.0
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
2019-07-18 09:36:55.697380: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
改用 CUDA_VISIBLE_DEVICES='1' 也无济于事。
Cuda (cuda_10.0.130_410.48_linux.run) 已安装。
$ cat /usr/local/cuda/version.txt
CUDA Version 10.0.130
CuDNN (cudnn-10.0-linux-x64-v7.4.2.24.tgz) 也是：
$ cat /usr/include/x86_64-linux-gnu/cudnn_v*.h | grep CUDNN_MAJOR -A 2 | head -n 3
#define CUDNN_MAJOR 6
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 21
TensorFlow (pip3 install tensorflow-gpu):
$ python3 -c 'import tensorflow as tf; print(tf.__version__)'
1.14.0
Nvidia 驱动程序（NVIDIA-Linux-x86_64-410.73.run）还：
$ nvidia-smi
Thu Jul 18 09:35:03 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.73 Driver Version: 410.73 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1070 Off | 00000000:01:00.0 On | N/A |
| 0% 45C P8 17W / 230W | 569MiB / 8111MiB | 19% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2270 G /usr/lib/xorg/Xorg 301MiB |
| 0 3021 G /opt/zoom/zoom 14MiB |
| 0 3503 G ...-token=CB875E52FAB2279C6A34C6519188AD9C 71MiB |
| 0 3534 G ...uest-channel-token=16121978823314344450 56MiB |
| 0 3618 G ...uest-channel-token=12369473663213430887 52MiB |
| 0 4249 G ...uest-channel-token=13759302641460814281 62MiB |
| 0 4499 G ...uest-channel-token=10576172133955227583 7MiB |
+-----------------------------------------------------------------------------+
我正在使用 Linux Mint 18.2。
有什么想法吗？
解决了。我卸载了从 NVIDIA-Linux-x86_64-410.73.run 安装的突触包管理器中显示的所有 Nvidia 驱动程序版本，现在可以了。
备案：使用命令行卸载可能如下所示：
sudo nvidia-uninstall
sudo apt-get remove --purge nvidia-*
$ python3 -c 'import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))'
2019-07-18 10:57:07.020764: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-18 10:57:07.059271: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3312000000 Hz
2019-07-18 10:57:07.060038: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x53aec90 executing computations on platform Host. Devices:
2019-07-18 10:57:07.060060: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2019-07-18 10:57:07.069543: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-18 10:57:07.216124: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.216596: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54792a0 executing computations on platform CUDA. Devices:
2019-07-18 10:57:07.216612: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1
2019-07-18 10:57:07.216803: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.217224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
2019-07-18 10:57:07.218763: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-18 10:57:07.243155: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-18 10:57:07.257961: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-18 10:57:07.263297: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-18 10:57:07.298517: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-18 10:57:07.321558: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-18 10:57:07.394510: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-18 10:57:07.394806: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.396131: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.397206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-18 10:57:07.397798: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-18 10:57:07.400997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-18 10:57:07.401041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-07-18 10:57:07.401059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-07-18 10:57:07.401572: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.402874: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.404129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7060 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1
2019-07-18 10:57:07.405492: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1
启动 TensorFlow 会话时，未检测到 GPU (CUDA_ERROR_INVALID_DEVICE: invalid device ordinal)：
$ CUDA_VISIBLE_DEVICES='0' python3 -c 'import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))'
2019-07-18 09:36:55.661519: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-18 09:36:55.684438: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3312000000 Hz
2019-07-18 09:36:55.684721: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x41adbb0 executing computations on platform Host. Devices:
2019-07-18 09:36:55.684750: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2019-07-18 09:36:55.686513: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-18 09:36:55.696958: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
2019-07-18 09:36:55.697001: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: tobias-Z170-HD3P
2019-07-18 09:36:55.697006: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: tobias-Z170-HD3P
2019-07-18 09:36:55.697084: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 410.73.0
2019-07-18 09:36:55.697108: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 410.73.0
2019-07-18 09:36:55.697113: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 410.73.0
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
2019-07-18 09:36:55.697380: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
改用 CUDA_VISIBLE_DEVICES='1' 也无济于事。
Cuda (cuda_10.0.130_410.48_linux.run) 已安装。
$ cat /usr/local/cuda/version.txt
CUDA Version 10.0.130
CuDNN (cudnn-10.0-linux-x64-v7.4.2.24.tgz) 也是：
$ cat /usr/include/x86_64-linux-gnu/cudnn_v*.h | grep CUDNN_MAJOR -A 2 | head -n 3
#define CUDNN_MAJOR 6
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 21
TensorFlow (pip3 install tensorflow-gpu):
$ python3 -c 'import tensorflow as tf; print(tf.__version__)'
1.14.0
Nvidia 驱动程序（NVIDIA-Linux-x86_64-410.73.run）还：
$ nvidia-smi
Thu Jul 18 09:35:03 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.73 Driver Version: 410.73 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1070 Off | 00000000:01:00.0 On | N/A |
| 0% 45C P8 17W / 230W | 569MiB / 8111MiB | 19% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2270 G /usr/lib/xorg/Xorg 301MiB |
| 0 3021 G /opt/zoom/zoom 14MiB |
| 0 3503 G ...-token=CB875E52FAB2279C6A34C6519188AD9C 71MiB |
| 0 3534 G ...uest-channel-token=16121978823314344450 56MiB |
| 0 3618 G ...uest-channel-token=12369473663213430887 52MiB |
| 0 4249 G ...uest-channel-token=13759302641460814281 62MiB |
| 0 4499 G ...uest-channel-token=10576172133955227583 7MiB |
+-----------------------------------------------------------------------------+
我正在使用 Linux Mint 18.2。
有什么想法吗？
解决了。我卸载了从 NVIDIA-Linux-x86_64-410.73.run 安装的突触包管理器中显示的所有 Nvidia 驱动程序版本，现在可以了。
备案：使用命令行卸载可能如下所示：
sudo nvidia-uninstall
sudo apt-get remove --purge nvidia-*
$ python3 -c 'import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))'
2019-07-18 10:57:07.020764: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-18 10:57:07.059271: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3312000000 Hz
2019-07-18 10:57:07.060038: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x53aec90 executing computations on platform Host. Devices:
2019-07-18 10:57:07.060060: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2019-07-18 10:57:07.069543: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-18 10:57:07.216124: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.216596: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54792a0 executing computations on platform CUDA. Devices:
2019-07-18 10:57:07.216612: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1
2019-07-18 10:57:07.216803: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.217224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
2019-07-18 10:57:07.218763: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-18 10:57:07.243155: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-18 10:57:07.257961: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-18 10:57:07.263297: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-18 10:57:07.298517: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-18 10:57:07.321558: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-18 10:57:07.394510: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-18 10:57:07.394806: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.396131: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.397206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-18 10:57:07.397798: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-18 10:57:07.400997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-18 10:57:07.401041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-07-18 10:57:07.401059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-07-18 10:57:07.401572: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.402874: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.404129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7060 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1
2019-07-18 10:57:07.405492: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1