使用 astropy.io 读取一堆 FITS 时出现 OSError 24(打开的文件太多)
OSError 24 (Too many open files) when reading bunch of FITS with astropy.io
我正在尝试使用 astropy.io.fits
:
将几个 2 000 FITS 加载到内存中
def readfits(filename):
with fits.open(filename) as ft:
# the fits contain a single HDU
data = ft[0].data
return data
data_sci = []
for i in range(2000):
data_sci.append(readfits("filename_{}.fits".format(i)))
但是,当到达第1015个文件时,会引发OSError: [Errno 24] Too many open
files
。
我有同样的问题:
def readfits(filename):
ft = fits.open(filename) as ft:
data = ft[0].data
ft.close()
return data
我怀疑 astropy.io.fits
没有正确关闭文件。有没有
我可以强制关闭文件的方法吗?
在查看了 astropy 文档后,我发现了这个:http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files
您的 readfits
函数实际上使文件句柄保持打开状态以保持对数据的访问,因为默认情况下它会创建一个 mmap to the data and does not read it entirely into physical memory, as explained: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files
顺便说一下,如果您只想要一个从第一个 HDU 中读取数据的函数,这已经内置:http://docs.astropy.org/en/v1.0.5/io/fits/api/files.html#astropy.io.fits.getdata
不需要重新发明轮子
只要你有记忆,你就可以调用这个函数并存储它的输出。我认为值得明确提及答案,但归功于 Iguananaut, bkaf, and this page.
def get_single_fits_data(fits_dir):
hdul = fits.open(fits_dir)
for hdu in hdul:
image_data = hdu.data.copy()
hdul.close()
gc.collect()
return image_data
我正在尝试使用 astropy.io.fits
:
def readfits(filename):
with fits.open(filename) as ft:
# the fits contain a single HDU
data = ft[0].data
return data
data_sci = []
for i in range(2000):
data_sci.append(readfits("filename_{}.fits".format(i)))
但是,当到达第1015个文件时,会引发OSError: [Errno 24] Too many open
files
。
我有同样的问题:
def readfits(filename):
ft = fits.open(filename) as ft:
data = ft[0].data
ft.close()
return data
我怀疑 astropy.io.fits
没有正确关闭文件。有没有
我可以强制关闭文件的方法吗?
在查看了 astropy 文档后,我发现了这个:http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files
您的 readfits
函数实际上使文件句柄保持打开状态以保持对数据的访问,因为默认情况下它会创建一个 mmap to the data and does not read it entirely into physical memory, as explained: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files
顺便说一下,如果您只想要一个从第一个 HDU 中读取数据的函数,这已经内置:http://docs.astropy.org/en/v1.0.5/io/fits/api/files.html#astropy.io.fits.getdata
不需要重新发明轮子
只要你有记忆,你就可以调用这个函数并存储它的输出。我认为值得明确提及答案,但归功于 Iguananaut, bkaf, and this page.
def get_single_fits_data(fits_dir):
hdul = fits.open(fits_dir)
for hdu in hdul:
image_data = hdu.data.copy()
hdul.close()
gc.collect()
return image_data