查找包含图像的子文件夹

Question

获取包含文件的子文件夹路径的最有效方法是什么。例如，如果这是我的输入结构。

inputFolder    
│
└───subFolder1
│   │
│   └───subfolder11
│       │   file1.jpg
│       │   file2.jpg
│       │   ...
│   
└───folder2
    │   file021.jpg
    │   file022.jpg

如果我通过 getFolders(inputPath)，它应该 return 输出为包含图像的文件夹列表 ['inputFolder/subFolder1/subFolder11','inputFolder/folder2']

目前我正在使用我的库 TreeHandler，它只是 os.walk 的包装器来获取所有文件。

import os
from treeHandler import treeHandler
th=treeHandler()
tempImageList=th.getFiles(path,['jpg'])
### basically tempImageList will be list of path of all files with '.jpg' extension

### now is the filtering part,the line which requires optimisation.
subFolderList=list(set(list(map(lambda x:os.path.join(*x.split('/')[:-1]),tempImageList))))

我认为它可以更有效地完成。

提前致谢

Answer 1

import os
import glob

original_path = './inputFolder/'

def get_subfolders(path):
    return [f.path for f in os.scandir(path) if f.is_dir()]

def get_files_in_subfolder(subfolder, extension):
    return glob.glob(subfolder + '/*' + extension)

files = []
subfolders = [original_path] + get_subfolders(original_path)
while len(subfolders) > 0:
    new_subfolder = subfolders.pop()
    print(new_subfolder)
    subfolders += get_subfolders(new_subfolder)
    files += get_files_in_subfolder(new_subfolder, '.jpg')

Answer 2

拆分路径的所有部分并重新加入它们似乎会降低效率。

查找“/”的最后一个实例的索引并进行切片工作得更快。

def remove_tail(path):
    index = path.rfind('/') # returns index of last appearance of '/' or -1 if not present
    return (path[:index] if index != -1  else '.') # return . for parent directory
.
.
.
subFolderList = list(set([remove_tail(path) for path in tempImageList]))

已在 AWA2 数据集文件夹（50 个文件夹和 37,322 张图像）上验证。
观察到结果快了大约 3 倍。
使用列表理解增强了可读性。
处理了父目录有图像的情况（这会导致现有实现出错）

添加用于验证的代码

import os
from treeHandler import treeHandler
import time

def remove_tail(path):
    index = path.rfind('/')
    return (path[:index] if index != -1  else '.')

th=treeHandler()
tempImageList= th.getFiles('JPEGImages',['jpg'])
tempImageList = tempImageList
### basically tempImageList will be list of path of all files with '.jpg' extension

### now is the filtering part,the line which requires optimisation.
print(len(tempImageList))
start = time.time()
originalSubFolderList=list(set(list(map(lambda x:os.path.join(*x.split('/')[:-1]),tempImageList))))
print("Current method takes", time.time() - start)

start = time.time()
newSubFolderList = list(set([remove_tail(path) for path in tempImageList]))
print("New method takes", time.time() - start)

print("Is outputs matching: ", originalSubFolderList == newSubFolderList)

查找包含图像的子文件夹

find sub folders which contain images

python

optimization

python-os