Why do I get "_curses.error: cbreak() returned ERR" when using TensorFlow CLI Debugger?

Question

我正在尝试使用 TensorFlow CLI 调试器来识别在网络训练期间导致 NaN 的操作，但是当我尝试运行代码时出现错误：

_curses.error: cbreak() returned ERR

我运行正在 Ubuntu 服务器上编写代码，我通过 SSH 连接到该服务器，并尝试遵循 this tutorial.

我曾尝试使用 tf.add_check_numerics_ops()，但网络中的层包含 while 循环，因此不兼容。这是引发错误的代码部分：

import tensorflow as tf
from tensorflow.python import debug as tf_debug
...
#Prepare data
train_data, val_data, test_data = dataset.prepare_datasets(model_config)

sess = tf.Session()
sess = tf_debug.LocalCLIDebugWrapperSession(sess)

# Create iterators
handle = tf.placeholder(tf.string, shape=[])
iterator = tf.data.Iterator.from_string_handle(handle, train_data.output_types, train_data.output_shapes)
mixed_spec, voice_spec, mixed_audio, voice_audio = iterator.get_next()

training_iterator = train_data.make_initializable_iterator()
validation_iterator = val_data.make_initializable_iterator()
testing_iterator = test_data.make_initializable_iterator()

training_handle = sess.run(training_iterator.string_handle())
...

完整的错误是：

Traceback (most recent call last):
  File "main.py", line 64, in <module>
    @ex.automain
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/experiment.py", line 137, in automain
    self.run_commandline()
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/experiment.py", line 260, in run_commandline
    return self.run(cmd_name, config_updates, named_configs, {}, args)
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/experiment.py", line 209, in run
    run()
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/run.py", line 221, in __call__
    self.result = self.main_function(*args)
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/config/captured_function.py", line 46, in captured_function
    result = wrapped(*args, **kwargs)
  File "main.py", line 95, in do_experiment
    training_handle = sess.run(training_iterator.string_handle())
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/wrappers/framework.py", line 455, in run
    is_callable_runner=bool(callable_runner)))
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/wrappers/local_cli_wrapper.py", line 255, in on_run_start
    self._run_start_response = self._launch_cli()
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/wrappers/local_cli_wrapper.py", line 431, in _launch_cli
    title_color=self._title_color)
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/cli/curses_ui.py", line 492, in run_ui
    self._screen_launch(enable_mouse_on_start=enable_mouse_on_start)
  File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/cli/curses_ui.py", line 445, in _screen_launch
    curses.cbreak()
_curses.error: cbreak() returned ERR

我对使用 Ubuntu（和 TensorFlow）还很陌生，但据我所知服务器确实安装了 ncurses，它应该允许所需的基于 curses 的界面：

acvn728@america:~/MScFinalProject$ dpkg -l '*ncurses*' | grep '^ii'
ii  libncurses5:amd64  6.0+20160213-1ubuntu1 amd64        shared libraries for terminal handling
ii  libncursesw5:amd64 6.0+20160213-1ubuntu1 amd64        shared libraries for terminal handling (wide character support)
ii  ncurses-base       6.0+20160213-1ubuntu1 all          basic terminal type definitions
ii  ncurses-bin        6.0+20160213-1ubuntu1 amd64        terminal-related programs and man pages
ii  ncurses-term       6.0+20160213-1ubuntu1 all          additional terminal type definitions

Answer 1

cbreak 会 return ERR 如果你运行一个不在 真正的终端（即与POSIX termios calls一起工作的东西）。

根据描述，

but the layers in the network include while loops so are not compatible

您似乎运行正在终端机中。

Answer 2

问题解决了！解决方案是改变

sess = tf_debug.LocalCLIDebugWrapperSession(sess)

至

sess = tf_debug.LocalCLIDebugWrapperSession(sess, ui_type="readline")

这类似于的解决方案，但我认为重要的是要注意它们的不同，因为 a) 它指的是不同的函数和不同的 API 和 b)正如该解决方案中所述，我并没有尝试从 IDE 运行。

Why do I get "_curses.error: cbreak() returned ERR" when using TensorFlow CLI Debugger?

Why do I get "_curses.error: cbreak() returned ERR" when using TensorFlow CLI Debugger?

python

debugging

ncurses

tensorflow