Numpy 知道异常
Numpy savez exceptions
我最近发现了这个说法：
np.savez is not multi-process safe, because it always uses
gettempdir() + key + ".npy". So, if you're running the same script
over different data sets, you will roach your own data when one
process removes the /tmp/arr_0.npy of another process.
As luck would have it, I'm in that exact situation, so I have an
example error:
Traceback (most recent call last): File
"/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 99, in <module> main()
File "/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 90, in main
np.savez("%smask-t%0.2f" % (outfile,threshold), result)
File "/usr/lib/python2.5/site-packages/numpy/lib/io.py", line 230, in savez
zip.write(filename, arcname=fname)
File "/usr/lib/python2.5/zipfile.py", line 541, in write
st = os.stat(filename) OSError: [Errno 2] No such file or directory: '/tmp/arr_0.npy'
Fortunately, replacing tempfile.gettempdir() with tempfile.mkdtemp()
AND remembering to clean up at the end with os.rmdir(direc) is all it
takes to fix it as far as I can tell.
这到底是什么意思，我个人如何才能避免这种情况？ numpy 是否将写入 .npz 文件的数据存储在临时文件中，这些临时文件可能与其他脚本中的其他临时 .npz 文件一样命名？！
我有一些 运行 几个小时的科学实验，并通过 np.savez 保存它们的结果。使用 savez 保存的结果的目标路径不同，因此每个实验都有自己的结果路径。但是，它们的脚本本身位于同一目录中。
有趣的是，脚本有时会 运行 发生此错误前几个小时：
> Traceback (most recent call last): File
> "/work/var/slurmd/state.node348.d/job20832/slurm_script", line 53, in
> <module>
> E = Exp6_06() File
> "/work/experiments/s06/cs06_ex06.py", line
> 150, in __init__
> self.__start() File "/work/experiments/s06/cs06_ex06.py", line
> 374, in __start
> File "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py",
> line 600, in savez_compressed
> _savez(file, args, kwds, True) File "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py",
> line 630, in _savez
> fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy') File "/sw/env/openmpi/2Q4/lib/python2.7/tempfile.py",
> line 314, in mkstemp
> return _mkstemp_inner(dir, prefix, suffix, flags) File "/sw/env/openmpi/2Q4/lib/python2.7/tempfile.py",
> line 244, in _mkstemp_inner
> fd = _os.open(file, flags, 0600) OSError: [Errno 2] No such file or directory:
> '/work/tmp/node001.admin.2016-03-06-152506.fx092.27432/tmp5UulEz-numpy.npy'
使用的 numpy 版本: 1.10.4
您的错误发生在 mkstemp。引用建议改用 mkdtemp：
https://docs.python.org/2/library/tempfile.html
tempfile.mkstemp([suffix=''[, prefix='tmp'[, dir=None[, text=False]]]])
Creates a temporary file in the most secure manner possible. There are no race conditions in the file’s creation, assuming that the platform properly implements the os.O_EXCL flag for os.open(). The file is readable and writable only by the creating user ID. If the platform uses permission bits to indicate whether a file is executable, the file is executable by no one. The file descriptor is not inherited by child processes.
和
tempfile.mkdtemp([suffix=''[, prefix='tmp'[, dir=None]]])
Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory’s creation. The directory is readable, writable, and searchable only by the creating user ID.
The user of mkdtemp() is responsible for deleting the temporary directory and its contents when done with it.
引用的错误似乎是在创建临时文件并将它们收集到 zip 之后发生的。您的错误似乎是在创建其中一个临时文件时发生的。
我对这些功能的使用还不够多，无法理解其中的区别。
我们可能需要在 lib/npyio.py.
中研究 _savez
你在用什么OS？ Linux、mac、windows?
这是 _savez 的核心，现在在 lib/npyio.py 文件中：
def _savez(file, args, kwds, compress):
....
zip = zipfile_factory(file, mode="w", compression=compression)
# Stage arrays in a temporary file on disk, before writing to zip.
=> fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')
os.close(fd)
try:
for key, val in namedict.items():
fname = key + '.npy'
fid = open(tmpfile, 'wb')
try:
format.write_array(fid, np.asanyarray(val))
fid.close()
fid = None
=> zip.write(tmpfile, arcname=fname)
finally:
if fid:
fid.close()
finally:
os.remove(tmpfile)
zip.close()
当临时 'save' 文件添加到存档时，引用的错误发生在循环的末尾。当它获取临时目录和临时文件 (fd) 时，您的错误很早就发生了。请注意，丢弃打开的文件句柄，只使用名称（对每个数组重复）。
令人惊讶的是临时目录在 mkstemp 函数调用期间竟然消失了。它几乎感觉像是 openmpi 问题而不是 savez 问题。
我最近发现了这个说法：
np.savez is not multi-process safe, because it always uses gettempdir() + key + ".npy". So, if you're running the same script over different data sets, you will roach your own data when one process removes the /tmp/arr_0.npy of another process.
As luck would have it, I'm in that exact situation, so I have an example error:
Traceback (most recent call last): File "/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 99, in <module> main() File "/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 90, in main np.savez("%smask-t%0.2f" % (outfile,threshold), result) File "/usr/lib/python2.5/site-packages/numpy/lib/io.py", line 230, in savez zip.write(filename, arcname=fname) File "/usr/lib/python2.5/zipfile.py", line 541, in write st = os.stat(filename) OSError: [Errno 2] No such file or directory: '/tmp/arr_0.npy'Fortunately, replacing
tempfile.gettempdir()withtempfile.mkdtemp()AND remembering to clean up at the end withos.rmdir(direc)is all it takes to fix it as far as I can tell.
这到底是什么意思，我个人如何才能避免这种情况？ numpy 是否将写入 .npz 文件的数据存储在临时文件中，这些临时文件可能与其他脚本中的其他临时 .npz 文件一样命名？！
我有一些 运行 几个小时的科学实验，并通过 np.savez 保存它们的结果。使用 savez 保存的结果的目标路径不同，因此每个实验都有自己的结果路径。但是，它们的脚本本身位于同一目录中。
有趣的是，脚本有时会 运行 发生此错误前几个小时：
> Traceback (most recent call last): File
> "/work/var/slurmd/state.node348.d/job20832/slurm_script", line 53, in
> <module>
> E = Exp6_06() File
> "/work/experiments/s06/cs06_ex06.py", line
> 150, in __init__
> self.__start() File "/work/experiments/s06/cs06_ex06.py", line
> 374, in __start
> File "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py",
> line 600, in savez_compressed
> _savez(file, args, kwds, True) File "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py",
> line 630, in _savez
> fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy') File "/sw/env/openmpi/2Q4/lib/python2.7/tempfile.py",
> line 314, in mkstemp
> return _mkstemp_inner(dir, prefix, suffix, flags) File "/sw/env/openmpi/2Q4/lib/python2.7/tempfile.py",
> line 244, in _mkstemp_inner
> fd = _os.open(file, flags, 0600) OSError: [Errno 2] No such file or directory:
> '/work/tmp/node001.admin.2016-03-06-152506.fx092.27432/tmp5UulEz-numpy.npy'
使用的 numpy 版本: 1.10.4
您的错误发生在 mkstemp。引用建议改用 mkdtemp：
https://docs.python.org/2/library/tempfile.html
tempfile.mkstemp([suffix=''[, prefix='tmp'[, dir=None[, text=False]]]])
Creates a temporary file in the most secure manner possible. There are no race conditions in the file’s creation, assuming that the platform properly implements the os.O_EXCL flag for os.open(). The file is readable and writable only by the creating user ID. If the platform uses permission bits to indicate whether a file is executable, the file is executable by no one. The file descriptor is not inherited by child processes.
和
tempfile.mkdtemp([suffix=''[, prefix='tmp'[, dir=None]]])
Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory’s creation. The directory is readable, writable, and searchable only by the creating user ID.
The user of mkdtemp() is responsible for deleting the temporary directory and its contents when done with it.
引用的错误似乎是在创建临时文件并将它们收集到 zip 之后发生的。您的错误似乎是在创建其中一个临时文件时发生的。
我对这些功能的使用还不够多，无法理解其中的区别。
我们可能需要在 lib/npyio.py.
_savez
你在用什么OS？ Linux、mac、windows?
这是 _savez 的核心，现在在 lib/npyio.py 文件中：
def _savez(file, args, kwds, compress):
....
zip = zipfile_factory(file, mode="w", compression=compression)
# Stage arrays in a temporary file on disk, before writing to zip.
=> fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')
os.close(fd)
try:
for key, val in namedict.items():
fname = key + '.npy'
fid = open(tmpfile, 'wb')
try:
format.write_array(fid, np.asanyarray(val))
fid.close()
fid = None
=> zip.write(tmpfile, arcname=fname)
finally:
if fid:
fid.close()
finally:
os.remove(tmpfile)
zip.close()
当临时 'save' 文件添加到存档时，引用的错误发生在循环的末尾。当它获取临时目录和临时文件 (fd) 时，您的错误很早就发生了。请注意，丢弃打开的文件句柄，只使用名称（对每个数组重复）。
令人惊讶的是临时目录在 mkstemp 函数调用期间竟然消失了。它几乎感觉像是 openmpi 问题而不是 savez 问题。