经过多次训练后，CuDNN 在 TF 2.x 中崩溃
CuDNN crash in TF 2.x after many epochs of training
我目前对我的 tensorflow 项目越来越绝望。
安装 tensorflow 花了好几个小时，直到我发现 PyCharm、Python 3.7 和 TF 2.x 不知何故不兼容。现在是 运行，但经过多次训练后，我得到了一个非常不明确的 CuDNN 错误。
你知道我的代码是否错误或者是否有例如安装错误？你能给我一个方向吗？
我也没有找到任何具体的搜索内容。
我的设置 [括号里是我也试过的]:
- 硬件： i7-4790K，32GB 内存和 GeForce 2070 Super 8GB
- OS: Windows 10 64位
- Python: 3.6.8 [和 3.7（tf 安装失败）]
- IDE: PyCharm 2020.1.1 [和 2020.1]
- 驱动程序： 最新 "Studio" 驱动程序 442.92 [以及最新 "gaming" 驱动程序]
- CuDA: 10.1 + 此版本的最新 CuDNN dll [我也试过 10.2，但 tf 没有检测到它]
- TF: 2.2.0 RC4 [ 2.0.x 和 2.1.5] 通过 PyCharm 安装的所有软件包（因此是 pip）
此错误发生在训练约 3 小时后。在其他情况下（或网络的参数化）错误发生得更早。在这里您可以看到下面代码片段的完整输出：
C:\Users\Fhnx\.virtualenvs\Processing-TA9ofq3q\Scripts\python.exe C:/Users/Fhnx/.../playground/AI_Predictor_Test.py
2020-05-08 11:47:25.924424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Starting training sweep with Epochs: 10000, LRstart: 0.01, LRend: 5e-05
2020-05-08 11:47:27.887135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-05-08 11:47:27.912998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5
coreClock: 1.815GHz coreCount: 40 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-08 11:47:27.913212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-05-08 11:47:27.921203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-08 11:47:27.930115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-05-08 11:47:27.932760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-05-08 11:47:27.944938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-05-08 11:47:27.952321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-05-08 11:47:27.960042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-08 11:47:27.960698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-08 11:47:27.961058: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-05-08 11:47:27.969636: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2df4e1dcd00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-08 11:47:27.969831: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-08 11:47:27.970579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5
coreClock: 1.815GHz coreCount: 40 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-08 11:47:27.970964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-05-08 11:47:27.971208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-08 11:47:27.971389: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-05-08 11:47:27.971602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-05-08 11:47:27.971839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-05-08 11:47:27.972112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-05-08 11:47:27.972324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-08 11:47:27.973322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-08 11:47:28.530960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-08 11:47:28.531109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-05-08 11:47:28.531180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-05-08 11:47:28.532337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6213 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-05-08 11:47:28.534819: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2df7aeb31a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-08 11:47:28.534946: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2070 SUPER, Compute Capability 7.5
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 22)] 0
__________________________________________________________________________________________________
tf_op_layer_ExpandDims (TensorF [(None, 22, 1)] 0 input_1[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_6 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_9 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_12 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_15 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
gaussian_dropout (GaussianDropo (None, 22, 64) 0 dense[0][0]
__________________________________________________________________________________________________
gaussian_dropout_2 (GaussianDro (None, 22, 64) 0 dense_3[0][0]
__________________________________________________________________________________________________
gaussian_dropout_4 (GaussianDro (None, 22, 64) 0 dense_6[0][0]
__________________________________________________________________________________________________
gaussian_dropout_6 (GaussianDro (None, 22, 64) 0 dense_9[0][0]
__________________________________________________________________________________________________
gaussian_dropout_8 (GaussianDro (None, 22, 64) 0 dense_12[0][0]
__________________________________________________________________________________________________
gaussian_dropout_10 (GaussianDr (None, 22, 64) 0 dense_15[0][0]
__________________________________________________________________________________________________
bidirectional (Bidirectional) (None, 22, 16) 4672 gaussian_dropout[0][0]
__________________________________________________________________________________________________
bidirectional_2 (Bidirectional) (None, 22, 16) 4672 gaussian_dropout_2[0][0]
__________________________________________________________________________________________________
bidirectional_4 (Bidirectional) (None, 22, 16) 4672 gaussian_dropout_4[0][0]
__________________________________________________________________________________________________
bidirectional_6 (Bidirectional) (None, 22, 16) 4672 gaussian_dropout_6[0][0]
__________________________________________________________________________________________________
bidirectional_8 (Bidirectional) (None, 22, 16) 4672 gaussian_dropout_8[0][0]
__________________________________________________________________________________________________
bidirectional_10 (Bidirectional (None, 22, 16) 4672 gaussian_dropout_10[0][0]
__________________________________________________________________________________________________
bidirectional_1 (Bidirectional) (None, 22, 16) 1600 bidirectional[0][0]
__________________________________________________________________________________________________
bidirectional_3 (Bidirectional) (None, 22, 16) 1600 bidirectional_2[0][0]
__________________________________________________________________________________________________
bidirectional_5 (Bidirectional) (None, 22, 16) 1600 bidirectional_4[0][0]
__________________________________________________________________________________________________
bidirectional_7 (Bidirectional) (None, 22, 16) 1600 bidirectional_6[0][0]
__________________________________________________________________________________________________
bidirectional_9 (Bidirectional) (None, 22, 16) 1600 bidirectional_8[0][0]
__________________________________________________________________________________________________
bidirectional_11 (Bidirectional (None, 22, 16) 1600 bidirectional_10[0][0]
__________________________________________________________________________________________________
conv1d (Conv1D) (None, 20, 13) 1780 bidirectional_1[0][0]
__________________________________________________________________________________________________
conv1d_4 (Conv1D) (None, 20, 13) 1780 bidirectional_3[0][0]
__________________________________________________________________________________________________
conv1d_8 (Conv1D) (None, 20, 13) 1780 bidirectional_5[0][0]
__________________________________________________________________________________________________
conv1d_12 (Conv1D) (None, 20, 13) 1780 bidirectional_7[0][0]
__________________________________________________________________________________________________
conv1d_16 (Conv1D) (None, 20, 13) 1780 bidirectional_9[0][0]
__________________________________________________________________________________________________
conv1d_20 (Conv1D) (None, 20, 13) 1780 bidirectional_11[0][0]
__________________________________________________________________________________________________
conv1d_1 (Conv1D) (None, 20, 10) 1620 conv1d[0][0]
__________________________________________________________________________________________________
conv1d_5 (Conv1D) (None, 20, 10) 1620 conv1d_4[0][0]
__________________________________________________________________________________________________
conv1d_9 (Conv1D) (None, 20, 10) 1620 conv1d_8[0][0]
__________________________________________________________________________________________________
conv1d_13 (Conv1D) (None, 20, 10) 1620 conv1d_12[0][0]
__________________________________________________________________________________________________
conv1d_17 (Conv1D) (None, 20, 10) 1620 conv1d_16[0][0]
__________________________________________________________________________________________________
conv1d_21 (Conv1D) (None, 20, 10) 1620 conv1d_20[0][0]
__________________________________________________________________________________________________
conv1d_2 (Conv1D) (None, 20, 7) 1620 conv1d_1[0][0]
__________________________________________________________________________________________________
conv1d_6 (Conv1D) (None, 20, 7) 1620 conv1d_5[0][0]
__________________________________________________________________________________________________
conv1d_10 (Conv1D) (None, 20, 7) 1620 conv1d_9[0][0]
__________________________________________________________________________________________________
conv1d_14 (Conv1D) (None, 20, 7) 1620 conv1d_13[0][0]
__________________________________________________________________________________________________
conv1d_18 (Conv1D) (None, 20, 7) 1620 conv1d_17[0][0]
__________________________________________________________________________________________________
conv1d_22 (Conv1D) (None, 20, 7) 1620 conv1d_21[0][0]
__________________________________________________________________________________________________
conv1d_3 (Conv1D) (None, 20, 4) 1620 conv1d_2[0][0]
__________________________________________________________________________________________________
conv1d_7 (Conv1D) (None, 20, 4) 1620 conv1d_6[0][0]
__________________________________________________________________________________________________
conv1d_11 (Conv1D) (None, 20, 4) 1620 conv1d_10[0][0]
__________________________________________________________________________________________________
conv1d_15 (Conv1D) (None, 20, 4) 1620 conv1d_14[0][0]
__________________________________________________________________________________________________
conv1d_19 (Conv1D) (None, 20, 4) 1620 conv1d_18[0][0]
__________________________________________________________________________________________________
conv1d_23 (Conv1D) (None, 20, 4) 1620 conv1d_22[0][0]
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, 20, 4) 16 conv1d_3[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 20, 4) 16 conv1d_7[0][0]
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 20, 4) 16 conv1d_11[0][0]
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 20, 4) 16 conv1d_15[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 20, 4) 16 conv1d_19[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 20, 4) 16 conv1d_23[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 20, 128) 640 batch_normalization[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 20, 128) 640 batch_normalization_1[0][0]
__________________________________________________________________________________________________
dense_7 (Dense) (None, 20, 128) 640 batch_normalization_2[0][0]
__________________________________________________________________________________________________
dense_10 (Dense) (None, 20, 128) 640 batch_normalization_3[0][0]
__________________________________________________________________________________________________
dense_13 (Dense) (None, 20, 128) 640 batch_normalization_4[0][0]
__________________________________________________________________________________________________
dense_16 (Dense) (None, 20, 128) 640 batch_normalization_5[0][0]
__________________________________________________________________________________________________
gaussian_dropout_1 (GaussianDro (None, 20, 128) 0 dense_1[0][0]
__________________________________________________________________________________________________
gaussian_dropout_3 (GaussianDro (None, 20, 128) 0 dense_4[0][0]
__________________________________________________________________________________________________
gaussian_dropout_5 (GaussianDro (None, 20, 128) 0 dense_7[0][0]
__________________________________________________________________________________________________
gaussian_dropout_7 (GaussianDro (None, 20, 128) 0 dense_10[0][0]
__________________________________________________________________________________________________
gaussian_dropout_9 (GaussianDro (None, 20, 128) 0 dense_13[0][0]
__________________________________________________________________________________________________
gaussian_dropout_11 (GaussianDr (None, 20, 128) 0 dense_16[0][0]
__________________________________________________________________________________________________
flatten (Flatten) (None, 2560) 0 gaussian_dropout_1[0][0]
__________________________________________________________________________________________________
flatten_1 (Flatten) (None, 2560) 0 gaussian_dropout_3[0][0]
__________________________________________________________________________________________________
flatten_2 (Flatten) (None, 2560) 0 gaussian_dropout_5[0][0]
__________________________________________________________________________________________________
flatten_3 (Flatten) (None, 2560) 0 gaussian_dropout_7[0][0]
__________________________________________________________________________________________________
flatten_4 (Flatten) (None, 2560) 0 gaussian_dropout_9[0][0]
__________________________________________________________________________________________________
flatten_5 (Flatten) (None, 2560) 0 gaussian_dropout_11[0][0]
__________________________________________________________________________________________________
dense_2 (Dense) (None, 1) 2561 flatten[0][0]
__________________________________________________________________________________________________
dense_5 (Dense) (None, 1) 2561 flatten_1[0][0]
__________________________________________________________________________________________________
dense_8 (Dense) (None, 1) 2561 flatten_2[0][0]
__________________________________________________________________________________________________
dense_11 (Dense) (None, 1) 2561 flatten_3[0][0]
__________________________________________________________________________________________________
dense_14 (Dense) (None, 1) 2561 flatten_4[0][0]
__________________________________________________________________________________________________
dense_17 (Dense) (None, 1) 2561 flatten_5[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate) (None, 6) 0 dense_2[0][0]
dense_5[0][0]
dense_8[0][0]
dense_11[0][0]
dense_14[0][0]
dense_17[0][0]
==================================================================================================
Total params: 97,542
Trainable params: 97,494
Non-trainable params: 48
__________________________________________________________________________________________________
***** Training Net ForkedConvLSTM_D64_LSTM2x8_Conv4x20x4_D1x128_dr0.40 now *****
BatchSize: 2108, NumNetParams: 97542, Feature shape: (500000, 22), Output shape: (500000, 6), In/Out Elem.: 14.0000M with est. size: 448.0000 MB
Epoch 1/10000
2020-05-08 11:47:57.675309: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-08 11:47:57.962354: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-08 11:47:59.216097: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
238/238 [==============================] - 21s 90ms/step - loss: 0.3145 - val_loss: 0.0846 - lr: 0.0100
Epoch 2/10000
238/238 [==============================] - 15s 62ms/step - loss: 0.0851 - val_loss: 0.0837 - lr: 0.0100
[...]
Epoch 694/10000
238/238 [==============================] - 14s 61ms/step - loss: 0.0833 - val_loss: 0.0836 - lr: 5.0000e-05
Epoch 695/10000
6/238 [..............................] - ETA: 12s - loss: 0.08302020-05-08 14:39:02.141015: E tensorflow/stream_executor/dnn.cc:613] CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1986): 'cudnnRNNBackwardData( cudnn.handle(), rnn_desc.handle(), model_dims.max_seq_length, output_desc.handles(), output_data.opaque(), output_desc.handles(), output_backprop_data.opaque(), output_h_desc.handle(), output_h_backprop_data.opaque(), output_c_desc.handle(), output_c_backprop_data.opaque(), rnn_desc.params_handle(), params.opaque(), input_h_desc.handle(), input_h_data.opaque(), input_c_desc.handle(), input_c_data.opaque(), input_desc.handles(), input_backprop_data->opaque(), input_h_desc.handle(), input_h_backprop_data->opaque(), input_c_desc.handle(), input_c_backprop_data->opaque(), workspace.opaque(), workspace.size(), reserve_space_data->opaque(), reserve_space_data->size())'
2020-05-08 14:39:02.141642: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cudnn_rnn_ops.cc:1922 : Internal: Failed to call ThenRnnBackward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 16, 8, 1, 22, 2108, 8]
2020-05-08 14:39:02.141037: F tensorflow/stream_executor/cuda/cuda_dnn.cc:189] Check failed: status == CUDNN_STATUS_SUCCESS (7 vs. 0)Failed to set cuDNN stream.
20
Process finished with exit code -1073740791 (0xC0000409)
下面是一些代码，应该能够 运行 并产生以上输出：
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# from os import environ
# environ['TF_CPP_MIN_LOG_LEVEL'] = '1'
from tensorflow.keras.models import *
from tensorflow.keras.layers import *
from tensorflow.keras.optimizers import *
import tensorflow as tf
import numpy as np
import sys
def build_model_simple(inputLength=1, outputLength=1, lr=0.0001, device="/gpu:0",
dropoutRate=0.4,
nNeuFirstDense=64,
numLSTM=2, nNeuLSTM=8,
numConv=4, nFiltConv=20, szConvKernel=4,
numDenseInner=1, nNeuDenseInner=128):
tf.keras.backend.set_floatx('float32')
with tf.device(device):
input = Input(shape=(inputLength,), dtype=tf.float32)
inputExp = tf.expand_dims(input, -1)
allInner = []
for _ in range(outputLength):
inner = Dense(nNeuFirstDense, activation="linear")(inputExp)
inner = GaussianDropout(rate=dropoutRate)(inner)
if numLSTM and nNeuLSTM:
for _ in range(numLSTM):
inner = (Bidirectional(LSTM(nNeuLSTM, return_sequences=True))(inner))
if numConv:
for _ in range(numConv):
inner = Conv1D(filters=nFiltConv, kernel_size=szConvKernel,
strides=1, padding='valid',
data_format='channels_first')(inner)
inner = BatchNormalization()(inner)
if numDenseInner:
for _ in range(numDenseInner):
inner = Dense(nNeuDenseInner, activation="linear")(inner)
inner = GaussianDropout(rate=dropoutRate)(inner)
inner = Flatten()(inner)
inner = Dense(1, activation="linear")(inner)
allInner.append(inner)
out = Concatenate()(allInner)
# out = outTmp * outTmp * outTmp
model = Model(inputs=input, outputs=out)
model.compile(loss="mse", optimizer=Adam(lr=lr))
# model.compile(loss="mse", optimizer=Adadelta())
return model, 'ForkedConvLSTM_D{}_LSTM{}x{}_Conv{}x{}x{}_D{}x{}_dr{:.2f}'.format(
nNeuFirstDense,
numLSTM, nNeuLSTM,
numConv, nFiltConv, szConvKernel,
numDenseInner, nNeuDenseInner,
dropoutRate)
def scheduler(epoch, lrStart, lrEnd, lrDecay=0.05, lrNStable=10):
lr = lrStart
if epoch > lrNStable:
fac = tf.math.exp(lrDecay * (lrNStable - epoch))
lr = lrStart * fac + lrEnd * (1 - fac)
return lr
if __name__ == '__main__':
numFeatures = 22
numOutputs = 6
trainIn = np.random.rand(500000, numFeatures)
trainOut = np.random.rand(500000, numOutputs)
valiIn = np.random.rand(12000, numFeatures)
valiOut = np.random.rand(12000, numOutputs)
numDataElements = trainIn.shape[0] * (trainIn.shape[1] + trainOut.shape[1])
sizeCalc = numDataElements * sys.getsizeof(trainIn[0][0])
EPOCHS = 10000
LEARNING_RATE_START = 0.01
LEARNING_RATE_END = 0.00005
LEARNING_DECAY = 0.05
print("Starting training sweep with Epochs: {}, LRstart: {}, LRend: {}".format(
EPOCHS, LEARNING_RATE_START, LEARNING_RATE_END))
network, nwName = build_model_simple(inputLength=numFeatures, outputLength=numOutputs)
netWeights = network.get_weights()
numNetPrams = np.sum([np.prod(ele.shape) for ele in netWeights])
# Estimation of Batch Size: GRAM * RAM Factor / NumParams in Net = ~75k. This divided by 30 for to get a
# good rough estimate for the batch size
BATCH_SIZE = int(np.floor(8 * 1e9 * 0.9 / numNetPrams / 35))
network.summary()
print("***** Training Net {} now *****".format(nwName))
print("BatchSize: {}, NumNetParams: {}, Feature shape: {}, Output shape: "
"{}, In/Out Elem.: {:.4f}M with est. size: {:.4f} MB".format(
BATCH_SIZE, numNetPrams, trainIn.shape, trainOut.shape,
numDataElements / 1e6, sizeCalc / 1e6))
callback = tf.keras.callbacks.LearningRateScheduler(
lambda x: scheduler(x, LEARNING_RATE_START, LEARNING_RATE_END, LEARNING_DECAY))
fitRes = network.fit(trainIn, trainOut, batch_size=BATCH_SIZE, epochs=EPOCHS,
validation_data=(valiIn, valiOut),
callbacks=[callback, tf.keras.callbacks.TerminateOnNaN()],
verbose=1)
logging.info("FINISHED")
关注我的人：
我玩了很多不同的版本。
我什至试图通过将新 dll 与旧名称进行符号链接来让 CUDA 10.2 工作。
但即使这样也没有修复错误。
我终于设法让它工作了，方法是删除所有 NVidia 内容（包括驱动程序）并使用此版本的工作室驱动程序安装最新的 10.1 版本（从 19 年底开始）。所以版本 431.86，而不是最新的工作室版本 441.66。
我不认为之前的安装有错误，因此我的估计是驱动版本一直是问题...
我目前对我的 tensorflow 项目越来越绝望。 安装 tensorflow 花了好几个小时，直到我发现 PyCharm、Python 3.7 和 TF 2.x 不知何故不兼容。现在是 运行，但经过多次训练后，我得到了一个非常不明确的 CuDNN 错误。 你知道我的代码是否错误或者是否有例如安装错误？你能给我一个方向吗？ 我也没有找到任何具体的搜索内容。
我的设置 [括号里是我也试过的]:
- 硬件： i7-4790K，32GB 内存和 GeForce 2070 Super 8GB
- OS: Windows 10 64位
- Python: 3.6.8 [和 3.7（tf 安装失败）]
- IDE: PyCharm 2020.1.1 [和 2020.1]
- 驱动程序： 最新 "Studio" 驱动程序 442.92 [以及最新 "gaming" 驱动程序]
- CuDA: 10.1 + 此版本的最新 CuDNN dll [我也试过 10.2，但 tf 没有检测到它]
- TF: 2.2.0 RC4 [ 2.0.x 和 2.1.5] 通过 PyCharm 安装的所有软件包（因此是 pip）
此错误发生在训练约 3 小时后。在其他情况下（或网络的参数化）错误发生得更早。在这里您可以看到下面代码片段的完整输出：
C:\Users\Fhnx\.virtualenvs\Processing-TA9ofq3q\Scripts\python.exe C:/Users/Fhnx/.../playground/AI_Predictor_Test.py
2020-05-08 11:47:25.924424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Starting training sweep with Epochs: 10000, LRstart: 0.01, LRend: 5e-05
2020-05-08 11:47:27.887135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-05-08 11:47:27.912998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5
coreClock: 1.815GHz coreCount: 40 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-08 11:47:27.913212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-05-08 11:47:27.921203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-08 11:47:27.930115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-05-08 11:47:27.932760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-05-08 11:47:27.944938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-05-08 11:47:27.952321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-05-08 11:47:27.960042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-08 11:47:27.960698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-08 11:47:27.961058: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-05-08 11:47:27.969636: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2df4e1dcd00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-08 11:47:27.969831: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-08 11:47:27.970579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5
coreClock: 1.815GHz coreCount: 40 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-08 11:47:27.970964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-05-08 11:47:27.971208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-08 11:47:27.971389: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-05-08 11:47:27.971602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-05-08 11:47:27.971839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-05-08 11:47:27.972112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-05-08 11:47:27.972324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-08 11:47:27.973322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-08 11:47:28.530960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-08 11:47:28.531109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-05-08 11:47:28.531180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-05-08 11:47:28.532337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6213 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-05-08 11:47:28.534819: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2df7aeb31a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-08 11:47:28.534946: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2070 SUPER, Compute Capability 7.5
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 22)] 0
__________________________________________________________________________________________________
tf_op_layer_ExpandDims (TensorF [(None, 22, 1)] 0 input_1[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_6 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_9 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_12 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
dense_15 (Dense) (None, 22, 64) 128 tf_op_layer_ExpandDims[0][0]
__________________________________________________________________________________________________
gaussian_dropout (GaussianDropo (None, 22, 64) 0 dense[0][0]
__________________________________________________________________________________________________
gaussian_dropout_2 (GaussianDro (None, 22, 64) 0 dense_3[0][0]
__________________________________________________________________________________________________
gaussian_dropout_4 (GaussianDro (None, 22, 64) 0 dense_6[0][0]
__________________________________________________________________________________________________
gaussian_dropout_6 (GaussianDro (None, 22, 64) 0 dense_9[0][0]
__________________________________________________________________________________________________
gaussian_dropout_8 (GaussianDro (None, 22, 64) 0 dense_12[0][0]
__________________________________________________________________________________________________
gaussian_dropout_10 (GaussianDr (None, 22, 64) 0 dense_15[0][0]
__________________________________________________________________________________________________
bidirectional (Bidirectional) (None, 22, 16) 4672 gaussian_dropout[0][0]
__________________________________________________________________________________________________
bidirectional_2 (Bidirectional) (None, 22, 16) 4672 gaussian_dropout_2[0][0]
__________________________________________________________________________________________________
bidirectional_4 (Bidirectional) (None, 22, 16) 4672 gaussian_dropout_4[0][0]
__________________________________________________________________________________________________
bidirectional_6 (Bidirectional) (None, 22, 16) 4672 gaussian_dropout_6[0][0]
__________________________________________________________________________________________________
bidirectional_8 (Bidirectional) (None, 22, 16) 4672 gaussian_dropout_8[0][0]
__________________________________________________________________________________________________
bidirectional_10 (Bidirectional (None, 22, 16) 4672 gaussian_dropout_10[0][0]
__________________________________________________________________________________________________
bidirectional_1 (Bidirectional) (None, 22, 16) 1600 bidirectional[0][0]
__________________________________________________________________________________________________
bidirectional_3 (Bidirectional) (None, 22, 16) 1600 bidirectional_2[0][0]
__________________________________________________________________________________________________
bidirectional_5 (Bidirectional) (None, 22, 16) 1600 bidirectional_4[0][0]
__________________________________________________________________________________________________
bidirectional_7 (Bidirectional) (None, 22, 16) 1600 bidirectional_6[0][0]
__________________________________________________________________________________________________
bidirectional_9 (Bidirectional) (None, 22, 16) 1600 bidirectional_8[0][0]
__________________________________________________________________________________________________
bidirectional_11 (Bidirectional (None, 22, 16) 1600 bidirectional_10[0][0]
__________________________________________________________________________________________________
conv1d (Conv1D) (None, 20, 13) 1780 bidirectional_1[0][0]
__________________________________________________________________________________________________
conv1d_4 (Conv1D) (None, 20, 13) 1780 bidirectional_3[0][0]
__________________________________________________________________________________________________
conv1d_8 (Conv1D) (None, 20, 13) 1780 bidirectional_5[0][0]
__________________________________________________________________________________________________
conv1d_12 (Conv1D) (None, 20, 13) 1780 bidirectional_7[0][0]
__________________________________________________________________________________________________
conv1d_16 (Conv1D) (None, 20, 13) 1780 bidirectional_9[0][0]
__________________________________________________________________________________________________
conv1d_20 (Conv1D) (None, 20, 13) 1780 bidirectional_11[0][0]
__________________________________________________________________________________________________
conv1d_1 (Conv1D) (None, 20, 10) 1620 conv1d[0][0]
__________________________________________________________________________________________________
conv1d_5 (Conv1D) (None, 20, 10) 1620 conv1d_4[0][0]
__________________________________________________________________________________________________
conv1d_9 (Conv1D) (None, 20, 10) 1620 conv1d_8[0][0]
__________________________________________________________________________________________________
conv1d_13 (Conv1D) (None, 20, 10) 1620 conv1d_12[0][0]
__________________________________________________________________________________________________
conv1d_17 (Conv1D) (None, 20, 10) 1620 conv1d_16[0][0]
__________________________________________________________________________________________________
conv1d_21 (Conv1D) (None, 20, 10) 1620 conv1d_20[0][0]
__________________________________________________________________________________________________
conv1d_2 (Conv1D) (None, 20, 7) 1620 conv1d_1[0][0]
__________________________________________________________________________________________________
conv1d_6 (Conv1D) (None, 20, 7) 1620 conv1d_5[0][0]
__________________________________________________________________________________________________
conv1d_10 (Conv1D) (None, 20, 7) 1620 conv1d_9[0][0]
__________________________________________________________________________________________________
conv1d_14 (Conv1D) (None, 20, 7) 1620 conv1d_13[0][0]
__________________________________________________________________________________________________
conv1d_18 (Conv1D) (None, 20, 7) 1620 conv1d_17[0][0]
__________________________________________________________________________________________________
conv1d_22 (Conv1D) (None, 20, 7) 1620 conv1d_21[0][0]
__________________________________________________________________________________________________
conv1d_3 (Conv1D) (None, 20, 4) 1620 conv1d_2[0][0]
__________________________________________________________________________________________________
conv1d_7 (Conv1D) (None, 20, 4) 1620 conv1d_6[0][0]
__________________________________________________________________________________________________
conv1d_11 (Conv1D) (None, 20, 4) 1620 conv1d_10[0][0]
__________________________________________________________________________________________________
conv1d_15 (Conv1D) (None, 20, 4) 1620 conv1d_14[0][0]
__________________________________________________________________________________________________
conv1d_19 (Conv1D) (None, 20, 4) 1620 conv1d_18[0][0]
__________________________________________________________________________________________________
conv1d_23 (Conv1D) (None, 20, 4) 1620 conv1d_22[0][0]
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, 20, 4) 16 conv1d_3[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 20, 4) 16 conv1d_7[0][0]
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 20, 4) 16 conv1d_11[0][0]
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 20, 4) 16 conv1d_15[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 20, 4) 16 conv1d_19[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 20, 4) 16 conv1d_23[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 20, 128) 640 batch_normalization[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 20, 128) 640 batch_normalization_1[0][0]
__________________________________________________________________________________________________
dense_7 (Dense) (None, 20, 128) 640 batch_normalization_2[0][0]
__________________________________________________________________________________________________
dense_10 (Dense) (None, 20, 128) 640 batch_normalization_3[0][0]
__________________________________________________________________________________________________
dense_13 (Dense) (None, 20, 128) 640 batch_normalization_4[0][0]
__________________________________________________________________________________________________
dense_16 (Dense) (None, 20, 128) 640 batch_normalization_5[0][0]
__________________________________________________________________________________________________
gaussian_dropout_1 (GaussianDro (None, 20, 128) 0 dense_1[0][0]
__________________________________________________________________________________________________
gaussian_dropout_3 (GaussianDro (None, 20, 128) 0 dense_4[0][0]
__________________________________________________________________________________________________
gaussian_dropout_5 (GaussianDro (None, 20, 128) 0 dense_7[0][0]
__________________________________________________________________________________________________
gaussian_dropout_7 (GaussianDro (None, 20, 128) 0 dense_10[0][0]
__________________________________________________________________________________________________
gaussian_dropout_9 (GaussianDro (None, 20, 128) 0 dense_13[0][0]
__________________________________________________________________________________________________
gaussian_dropout_11 (GaussianDr (None, 20, 128) 0 dense_16[0][0]
__________________________________________________________________________________________________
flatten (Flatten) (None, 2560) 0 gaussian_dropout_1[0][0]
__________________________________________________________________________________________________
flatten_1 (Flatten) (None, 2560) 0 gaussian_dropout_3[0][0]
__________________________________________________________________________________________________
flatten_2 (Flatten) (None, 2560) 0 gaussian_dropout_5[0][0]
__________________________________________________________________________________________________
flatten_3 (Flatten) (None, 2560) 0 gaussian_dropout_7[0][0]
__________________________________________________________________________________________________
flatten_4 (Flatten) (None, 2560) 0 gaussian_dropout_9[0][0]
__________________________________________________________________________________________________
flatten_5 (Flatten) (None, 2560) 0 gaussian_dropout_11[0][0]
__________________________________________________________________________________________________
dense_2 (Dense) (None, 1) 2561 flatten[0][0]
__________________________________________________________________________________________________
dense_5 (Dense) (None, 1) 2561 flatten_1[0][0]
__________________________________________________________________________________________________
dense_8 (Dense) (None, 1) 2561 flatten_2[0][0]
__________________________________________________________________________________________________
dense_11 (Dense) (None, 1) 2561 flatten_3[0][0]
__________________________________________________________________________________________________
dense_14 (Dense) (None, 1) 2561 flatten_4[0][0]
__________________________________________________________________________________________________
dense_17 (Dense) (None, 1) 2561 flatten_5[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate) (None, 6) 0 dense_2[0][0]
dense_5[0][0]
dense_8[0][0]
dense_11[0][0]
dense_14[0][0]
dense_17[0][0]
==================================================================================================
Total params: 97,542
Trainable params: 97,494
Non-trainable params: 48
__________________________________________________________________________________________________
***** Training Net ForkedConvLSTM_D64_LSTM2x8_Conv4x20x4_D1x128_dr0.40 now *****
BatchSize: 2108, NumNetParams: 97542, Feature shape: (500000, 22), Output shape: (500000, 6), In/Out Elem.: 14.0000M with est. size: 448.0000 MB
Epoch 1/10000
2020-05-08 11:47:57.675309: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-08 11:47:57.962354: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-08 11:47:59.216097: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
238/238 [==============================] - 21s 90ms/step - loss: 0.3145 - val_loss: 0.0846 - lr: 0.0100
Epoch 2/10000
238/238 [==============================] - 15s 62ms/step - loss: 0.0851 - val_loss: 0.0837 - lr: 0.0100
[...]
Epoch 694/10000
238/238 [==============================] - 14s 61ms/step - loss: 0.0833 - val_loss: 0.0836 - lr: 5.0000e-05
Epoch 695/10000
6/238 [..............................] - ETA: 12s - loss: 0.08302020-05-08 14:39:02.141015: E tensorflow/stream_executor/dnn.cc:613] CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1986): 'cudnnRNNBackwardData( cudnn.handle(), rnn_desc.handle(), model_dims.max_seq_length, output_desc.handles(), output_data.opaque(), output_desc.handles(), output_backprop_data.opaque(), output_h_desc.handle(), output_h_backprop_data.opaque(), output_c_desc.handle(), output_c_backprop_data.opaque(), rnn_desc.params_handle(), params.opaque(), input_h_desc.handle(), input_h_data.opaque(), input_c_desc.handle(), input_c_data.opaque(), input_desc.handles(), input_backprop_data->opaque(), input_h_desc.handle(), input_h_backprop_data->opaque(), input_c_desc.handle(), input_c_backprop_data->opaque(), workspace.opaque(), workspace.size(), reserve_space_data->opaque(), reserve_space_data->size())'
2020-05-08 14:39:02.141642: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cudnn_rnn_ops.cc:1922 : Internal: Failed to call ThenRnnBackward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 16, 8, 1, 22, 2108, 8]
2020-05-08 14:39:02.141037: F tensorflow/stream_executor/cuda/cuda_dnn.cc:189] Check failed: status == CUDNN_STATUS_SUCCESS (7 vs. 0)Failed to set cuDNN stream.
20
Process finished with exit code -1073740791 (0xC0000409)
下面是一些代码，应该能够 运行 并产生以上输出：
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# from os import environ
# environ['TF_CPP_MIN_LOG_LEVEL'] = '1'
from tensorflow.keras.models import *
from tensorflow.keras.layers import *
from tensorflow.keras.optimizers import *
import tensorflow as tf
import numpy as np
import sys
def build_model_simple(inputLength=1, outputLength=1, lr=0.0001, device="/gpu:0",
dropoutRate=0.4,
nNeuFirstDense=64,
numLSTM=2, nNeuLSTM=8,
numConv=4, nFiltConv=20, szConvKernel=4,
numDenseInner=1, nNeuDenseInner=128):
tf.keras.backend.set_floatx('float32')
with tf.device(device):
input = Input(shape=(inputLength,), dtype=tf.float32)
inputExp = tf.expand_dims(input, -1)
allInner = []
for _ in range(outputLength):
inner = Dense(nNeuFirstDense, activation="linear")(inputExp)
inner = GaussianDropout(rate=dropoutRate)(inner)
if numLSTM and nNeuLSTM:
for _ in range(numLSTM):
inner = (Bidirectional(LSTM(nNeuLSTM, return_sequences=True))(inner))
if numConv:
for _ in range(numConv):
inner = Conv1D(filters=nFiltConv, kernel_size=szConvKernel,
strides=1, padding='valid',
data_format='channels_first')(inner)
inner = BatchNormalization()(inner)
if numDenseInner:
for _ in range(numDenseInner):
inner = Dense(nNeuDenseInner, activation="linear")(inner)
inner = GaussianDropout(rate=dropoutRate)(inner)
inner = Flatten()(inner)
inner = Dense(1, activation="linear")(inner)
allInner.append(inner)
out = Concatenate()(allInner)
# out = outTmp * outTmp * outTmp
model = Model(inputs=input, outputs=out)
model.compile(loss="mse", optimizer=Adam(lr=lr))
# model.compile(loss="mse", optimizer=Adadelta())
return model, 'ForkedConvLSTM_D{}_LSTM{}x{}_Conv{}x{}x{}_D{}x{}_dr{:.2f}'.format(
nNeuFirstDense,
numLSTM, nNeuLSTM,
numConv, nFiltConv, szConvKernel,
numDenseInner, nNeuDenseInner,
dropoutRate)
def scheduler(epoch, lrStart, lrEnd, lrDecay=0.05, lrNStable=10):
lr = lrStart
if epoch > lrNStable:
fac = tf.math.exp(lrDecay * (lrNStable - epoch))
lr = lrStart * fac + lrEnd * (1 - fac)
return lr
if __name__ == '__main__':
numFeatures = 22
numOutputs = 6
trainIn = np.random.rand(500000, numFeatures)
trainOut = np.random.rand(500000, numOutputs)
valiIn = np.random.rand(12000, numFeatures)
valiOut = np.random.rand(12000, numOutputs)
numDataElements = trainIn.shape[0] * (trainIn.shape[1] + trainOut.shape[1])
sizeCalc = numDataElements * sys.getsizeof(trainIn[0][0])
EPOCHS = 10000
LEARNING_RATE_START = 0.01
LEARNING_RATE_END = 0.00005
LEARNING_DECAY = 0.05
print("Starting training sweep with Epochs: {}, LRstart: {}, LRend: {}".format(
EPOCHS, LEARNING_RATE_START, LEARNING_RATE_END))
network, nwName = build_model_simple(inputLength=numFeatures, outputLength=numOutputs)
netWeights = network.get_weights()
numNetPrams = np.sum([np.prod(ele.shape) for ele in netWeights])
# Estimation of Batch Size: GRAM * RAM Factor / NumParams in Net = ~75k. This divided by 30 for to get a
# good rough estimate for the batch size
BATCH_SIZE = int(np.floor(8 * 1e9 * 0.9 / numNetPrams / 35))
network.summary()
print("***** Training Net {} now *****".format(nwName))
print("BatchSize: {}, NumNetParams: {}, Feature shape: {}, Output shape: "
"{}, In/Out Elem.: {:.4f}M with est. size: {:.4f} MB".format(
BATCH_SIZE, numNetPrams, trainIn.shape, trainOut.shape,
numDataElements / 1e6, sizeCalc / 1e6))
callback = tf.keras.callbacks.LearningRateScheduler(
lambda x: scheduler(x, LEARNING_RATE_START, LEARNING_RATE_END, LEARNING_DECAY))
fitRes = network.fit(trainIn, trainOut, batch_size=BATCH_SIZE, epochs=EPOCHS,
validation_data=(valiIn, valiOut),
callbacks=[callback, tf.keras.callbacks.TerminateOnNaN()],
verbose=1)
logging.info("FINISHED")
关注我的人：
我玩了很多不同的版本。 我什至试图通过将新 dll 与旧名称进行符号链接来让 CUDA 10.2 工作。 但即使这样也没有修复错误。
我终于设法让它工作了，方法是删除所有 NVidia 内容（包括驱动程序）并使用此版本的工作室驱动程序安装最新的 10.1 版本（从 19 年底开始）。所以版本 431.86，而不是最新的工作室版本 441.66。
我不认为之前的安装有错误，因此我的估计是驱动版本一直是问题...