如何使用多处理将 CSV 文件转换为 GZIP 文件。游泳池 Python

Question

我正在使用 gzip 将 csv 文件转换为 gunzip file.I 我正在尝试实施多处理以使过程更快。但是当我运行我的代码时，它甚至没有给我输出，它一直在运行ning 并且也没有给出错误。下面是我的代码

from multiprocessing import Pool
import gzip
path= r'D:\All Files'
def convertToGzip(file):
    file_ = os.path.join(path, file)
    print('File {} is being Converted'.format(file_))
    with open(os.path.join(path, file_), 'rb') as f_in:
        with gzip.open(file+'.gz', 'wb') as f_out:
            shutil.copyfileobj(f_in, f_out)
            
a_pool = Pool(processes=4)
result = a_pool.map(convertToGzip, os.listdir(path))

我不明白这里的问题在哪里，因为我对多处理了解不多

Answer 1

你可以试试这个：

import gzip
import os
import shutil

from multiprocessing import Pool

PATH = "D:\All Files"


def convert_to_gzip(fname):
    fpath = os.path.join(PATH, fname)

    print("File {} is being Converted".format(fpath))

    with open(fpath, "rb") as f_in:
        with gzip.open(fpath + ".gz", "wb") as f_out:
            shutil.copyfileobj(f_in, f_out)


def main():
    with Pool() as pool:
        result = pool.map(convert_to_gzip, os.listdir(PATH))


if __name__ == "__main__":
    main()

如何使用多处理将 CSV 文件转换为 GZIP 文件。游泳池 Python

how to Convert CSV file to GZIP file with use of multiprocessing. Pool in Python

python-3.x

csv

multiprocessing

gzip