为什么我的 Pytorch 张量大小会在一些批次后发生变化并包含 NaN?
Why does my Pytorch tensor size change and contain NaNs after some batches?
我正在训练 Pytorch 模型。一段时间后,即使在 shuffle 中,除了一些有限的张量外,模型只包含 NaN 值:
tensor([[[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan],
...,
[ 1.4641, 0.0360, -1.1528, ..., -2.3592, -2.6310, 6.3893],
[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan]]],
device='cuda:0', grad_fn=<AddBackward0>)
detect_anomaly函数return:
File "TestDownload.py", line 701, in <module>
main(learning_rate, batch_size, epochs, experiment)
File "TestDownload.py", line 635, in main
train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
File "TestDownload.py", line 486, in train
output = F.log_softmax(output, dim=2)
File "\lib\site-packages\torch\nn\functional.py", line 1672, in log_softmax
ret = input.log_softmax(dim)
(function _print_stack) Traceback (most recent call last):
File "TestDownload.py", line 701, in <module>
main(learning_rate, batch_size, epochs, experiment)
File "TestDownload.py", line 635, in main
train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
File "TestDownload.py", line 490, in train
loss.backward()
File "\lib\site-packages\comet_ml\monkey_patching.py", line 317, in wrapper
return_value = original(*args, **kwargs)
File "\lib\site-packages\torch\tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "\lib\site-packages\torch\autograd\__init__.py", line 145, in backward
Variable._execution_engine.run_backward(
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
参考下一行output = F.log_softmax(output, dim=2)
如果我只是用 try-except 来做,它会显示另一个错误:(当损失函数在包含 NaN 的张量上 运行ning 时)
[W ..\torch\csrc\autograd\python_anomaly_mode.cpp:104] Warning: Error detected in CtcLossBackward. Traceback of forward call that caused the error:
File "TestDownload.py", line 734, in <module>
# In[ ]:
File "TestDownload.py", line 667, in main
test(model, device, test_loader, criterion, epoch, iter_meter, experiment)
File "TestDownload.py", line 517, in train
loss.backward()
File "\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "\lib\site-packages\torch\nn\modules\loss.py", line 1590, in forward
return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,
File "\lib\site-packages\torch\nn\functional.py", line 2307, in ctc_loss
return torch.ctc_loss(
(function _print_stack)
Traceback (most recent call last):
File "TestDownload.py", line 518, in train
File "\lib\site-packages\comet_ml\monkey_patching.py", line 317, in wrapper
return_value = original(*args, **kwargs)
File "\lib\site-packages\torch\tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "\lib\site-packages\torch\autograd\__init__.py", line 145, in backward
Variable._execution_engine.run_backward(
RuntimeError: Function 'CtcLossBackward' returned nan values in its 0th output.
一个正常的张量应该是这样的:
tensor([[[-3.3904, -3.4340, -3.3703, ..., -3.3613, -3.5098, -3.4344]],
[[-3.3760, -3.2948, -3.2673, ..., -3.4039, -3.3827, -3.3919]],
[[-3.3857, -3.3358, -3.3901, ..., -3.4686, -3.4749, -3.3826]],
...,
[[-3.3568, -3.3502, -3.4416, ..., -3.4463, -3.4921, -3.3769]],
[[-3.4379, -3.3508, -3.3610, ..., -3.3707, -3.4030, -3.4244]],
[[-3.3919, -3.4513, -3.3565, ..., -3.2714, -3.3984, -3.3643]]],
device='cuda:0', grad_fn=<TransposeBackward0>)
请注意双括号,如果它们是导入的。
代码:
for batch_idx, _data in enumerate(train_loader):
spectrograms, labels, input_lengths, label_lengths = _data
spectrograms, labels = spectrograms.to(device), labels.to(device)
optimizer.zero_grad()
output = model(spectrograms)
output = F.log_softmax(output, dim=2)
output = output.transpose(0, 1) # (time, batch, n_class) # X, 1, 29
loss = criterion(output, labels, input_lengths, label_lengths)
loss.backward()
optimizer.step()
scheduler.step()
iter_meter.step()
此外,我尝试 运行 使用更大的批次大小(当前批次 size:1,更大的批次大小:6)并且它 运行 没有错误,直到 40%我遇到此错误的第一个纪元。
Cuda run out of memory
此外,我尝试对数据进行标准化 torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_mels=128, normalized=True)
将学习率从 5e-4 降低到 5e-5 也无济于事。
附加信息:我的数据集包含近 300000 个 .wav 文件,错误发生在第一个时期的 3-10%运行时间。
感谢任何提示,我很乐意提交更多信息。
错误源可能是损坏的输入或标签,其中包含 NaN 的 inf 值。
您可以使用
检查张量中是否没有 NaN 值
torch.isnan(tensor).any()
或者张量中的所有值既不是 inf
也不是 NaN
和
torch.isfinite(tensor).all()
我正在训练 Pytorch 模型。一段时间后,即使在 shuffle 中,除了一些有限的张量外,模型只包含 NaN 值:
tensor([[[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan],
...,
[ 1.4641, 0.0360, -1.1528, ..., -2.3592, -2.6310, 6.3893],
[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan]]],
device='cuda:0', grad_fn=<AddBackward0>)
detect_anomaly函数return:
File "TestDownload.py", line 701, in <module>
main(learning_rate, batch_size, epochs, experiment)
File "TestDownload.py", line 635, in main
train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
File "TestDownload.py", line 486, in train
output = F.log_softmax(output, dim=2)
File "\lib\site-packages\torch\nn\functional.py", line 1672, in log_softmax
ret = input.log_softmax(dim)
(function _print_stack) Traceback (most recent call last):
File "TestDownload.py", line 701, in <module>
main(learning_rate, batch_size, epochs, experiment)
File "TestDownload.py", line 635, in main
train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
File "TestDownload.py", line 490, in train
loss.backward()
File "\lib\site-packages\comet_ml\monkey_patching.py", line 317, in wrapper
return_value = original(*args, **kwargs)
File "\lib\site-packages\torch\tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "\lib\site-packages\torch\autograd\__init__.py", line 145, in backward
Variable._execution_engine.run_backward(
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
参考下一行output = F.log_softmax(output, dim=2)
如果我只是用 try-except 来做,它会显示另一个错误:(当损失函数在包含 NaN 的张量上 运行ning 时)
[W ..\torch\csrc\autograd\python_anomaly_mode.cpp:104] Warning: Error detected in CtcLossBackward. Traceback of forward call that caused the error:
File "TestDownload.py", line 734, in <module>
# In[ ]:
File "TestDownload.py", line 667, in main
test(model, device, test_loader, criterion, epoch, iter_meter, experiment)
File "TestDownload.py", line 517, in train
loss.backward()
File "\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "\lib\site-packages\torch\nn\modules\loss.py", line 1590, in forward
return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,
File "\lib\site-packages\torch\nn\functional.py", line 2307, in ctc_loss
return torch.ctc_loss(
(function _print_stack)
Traceback (most recent call last):
File "TestDownload.py", line 518, in train
File "\lib\site-packages\comet_ml\monkey_patching.py", line 317, in wrapper
return_value = original(*args, **kwargs)
File "\lib\site-packages\torch\tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "\lib\site-packages\torch\autograd\__init__.py", line 145, in backward
Variable._execution_engine.run_backward(
RuntimeError: Function 'CtcLossBackward' returned nan values in its 0th output.
一个正常的张量应该是这样的:
tensor([[[-3.3904, -3.4340, -3.3703, ..., -3.3613, -3.5098, -3.4344]],
[[-3.3760, -3.2948, -3.2673, ..., -3.4039, -3.3827, -3.3919]],
[[-3.3857, -3.3358, -3.3901, ..., -3.4686, -3.4749, -3.3826]],
...,
[[-3.3568, -3.3502, -3.4416, ..., -3.4463, -3.4921, -3.3769]],
[[-3.4379, -3.3508, -3.3610, ..., -3.3707, -3.4030, -3.4244]],
[[-3.3919, -3.4513, -3.3565, ..., -3.2714, -3.3984, -3.3643]]],
device='cuda:0', grad_fn=<TransposeBackward0>)
请注意双括号,如果它们是导入的。
代码:
for batch_idx, _data in enumerate(train_loader):
spectrograms, labels, input_lengths, label_lengths = _data
spectrograms, labels = spectrograms.to(device), labels.to(device)
optimizer.zero_grad()
output = model(spectrograms)
output = F.log_softmax(output, dim=2)
output = output.transpose(0, 1) # (time, batch, n_class) # X, 1, 29
loss = criterion(output, labels, input_lengths, label_lengths)
loss.backward()
optimizer.step()
scheduler.step()
iter_meter.step()
此外,我尝试 运行 使用更大的批次大小(当前批次 size:1,更大的批次大小:6)并且它 运行 没有错误,直到 40%我遇到此错误的第一个纪元。
Cuda run out of memory
此外,我尝试对数据进行标准化 torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_mels=128, normalized=True)
将学习率从 5e-4 降低到 5e-5 也无济于事。
附加信息:我的数据集包含近 300000 个 .wav 文件,错误发生在第一个时期的 3-10%运行时间。
感谢任何提示,我很乐意提交更多信息。
错误源可能是损坏的输入或标签,其中包含 NaN 的 inf 值。 您可以使用
检查张量中是否没有 NaN 值torch.isnan(tensor).any()
或者张量中的所有值既不是 inf
也不是 NaN
和
torch.isfinite(tensor).all()