Python glob.glob(dir) 内存错误

Question

我在搜索包含数百万个文件的文件夹时遇到内存问题。有谁知道如何克服这种情况？有什么方法可以限制 glob 将搜索的文件数量吗？所以它可以分块执行吗？

Traceback (most recent call last):
  File "./lb2_lmanager", line 533, in <module>
   main(sys.argv[1:])
 File "./lb2_lmanager", line 318, in main
    matched = match_files(policy.directory, policy.file_patterns)
  File "./lb2_lmanager", line 32, in wrapper
    res = func(*args, **kwargs)
  File "./lb2_lmanager", line 380, in match_files
    listing = glob.glob(directory)
  File "/usr/lib/python2.6/glob.py", line 16, in glob
    return list(iglob(pathname))
  File "/usr/lib/python2.6/glob.py", line 43, in iglob
    yield os.path.join(dirname, name)
  File "/usr/lib/python2.6/posixpath.py", line 70, in join
   path += '/' + b
MemoryError

Answer 1

尝试使用 generators 而不是 lists。
了解什么是生成器this

import glob
dir_list = glob.iglob(YOUR_DIRECTORY)
for file in dir_list:
    print file

将 YOUR_DIRECTORY 更改为您要列出的目录。

Python glob.glob(dir) 内存错误

Python glob.glob(dir) Memory Error

python

posix

glob