CuDNN -- 状态未初始化 (Keras/TensorFlow + Nvidia P100 + Linux)

Question

我无法通过 keras+tensorflow-backend 将我的（工作中的）LSTM 模型转换为利用 CuDNN。我正在使用：

张量流 1.10.1
Tensorflow-GPU 1.10.1
凯拉斯 2.2.2
Cuda 9.2
CuDNN 7.2.1（非常确定）
NVIDIA P100 GPU（驱动程序 390.87）。

代码示例：

def build_lstm(num_neurons, dropout, recurent_dropout):
    model = Sequential()
    model.add(LSTM(num_neurons, input_shape=(12,1), dropout=dropout, recurrent_dropout=recurent_dropout, unroll=True))
    model.add(Dense(1))
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

def build_cudnnlstm(num_neurons, dropout, recurent_dropout):
    model = Sequential()
    model.add(CuDNNLSTM(num_neurons, input_shape=(12,1)))
    model.add(Dropout(dropout))
    model.add(Dense(1))
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

但是，当我将 build_cudnnlstm 换成 build_lstm 时，出现以下错误：

Epoch 1/5 2018-09-10 15:58:53.726819: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2018-09-10 15:58:54.001406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:17:00.0
totalMemory: 15.90GiB freeMemory: 15.61GiB
2018-09-10 15:58:54.001491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-10 15:58:54.475955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-10 15:58:54.476019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0
2018-09-10 15:58:54.476036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N
2018-09-10 15:58:54.476408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15123 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:17:00.0, compute capability: 6.0)
2018-09-10 15:58:55.098145: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2018-09-10 15:58:55.098409: E tensorflow/stream_executor/cuda/cuda_dnn.cc:360] Possibly insufficient driver version: 390.87.0
2018-09-10 15:58:55.098496: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at cudnn_rnn_ops.cc:1214 : Unknown: Fail to find the dnn implementation.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/python3.6/site-packages/keras/engine/training.py", line 1037, in fit
    validation_steps=validation_steps)
  File "/usr/local/lib64/python3.6/site-packages/keras/engine/training_arrays.py", line 199, in fit_loop
    outs = f(ins_batch)
  File "/usr/local/lib64/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2666, in __call__
    return self._call(inputs)
  File "/usr/local/lib64/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2636, in _call
    fetched = self._callable_fn(*array_vals)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1382, in __call__
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
         [[Node: cu_dnnlstm_1/CudnnRNN = CudnnRNN[T=DT_FLOAT, _class=["loc:@training/Adam/gradients/cu_dnnlstm_1/CudnnRNN_grad/CudnnRNNBackprop"], direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=87654321, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](cu_dnnlstm_1/transpose, cu_dnnlstm_1/ExpandDims_1, cu_dnnlstm_1/ExpandDims_1, cu_dnnlstm_1/concat_1)]]
         [[Node: loss/mul/_79 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_782_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

拟合过程中出现此错误：

model.fit(samples, targets_1q, epochs=epochs, shuffle=True, verbose=2)

非常感谢任何帮助！

Answer 1

查看我的笔记，我似乎曾经运行对此进行了纠正，并使用：

pip3 install --upgrade tensorflow

pip3 install --upgrade tensorflow-gpu

你的里程我会有所不同。

查看你的CUDNN版本很简单——你知道CUDA安装在哪里吗？如果是这样，只需查看您移入该目录的 CUDNN headers。

Answer 2

也许你应该升级驱动，我记得396.37是对应Cuda 9.2的版本

CuDNN -- 状态未初始化 (Keras/TensorFlow + Nvidia P100 + Linux)

CuDNN -- Status Not Intitialized (Keras/TensorFlow + Nvidia P100 + Linux)

tensorflow

keras

gpu

lstm

cudnn