使用 openpyxl 处理非常大的文件 python

Question

我有一个包含 11,000 行和 10 列的电子表格。我正在尝试复制包含选定列的每一行，每行添加附加信息并输出到 txt。

不幸的是，我遇到了非常糟糕的性能问题，文件在 100 行后开始卡顿并杀死我的处理器。有没有办法加快速度或使用更好的方法？我已经在使用 read_only=True 和 data_only=True

内存最密集的部分是遍历每个单元格：

for i in range(probeStart, lastRow+1):
    dataRow =""
    for j in range (1,col+2):
        dataRow = dataRow + str(sheet.cell(row=i, column=j).value)  + "\t"

    sigP = db.get(str(sheet.cell(row= i, column=1).value), "notfound") #my additional information 
    a = str(sheet.cell(row = i, column = max_column-1).value) +"\t" 
    b  = str(sheet.cell(row = i, column = max_column).value) + "\t"
    string1  = dataRow + a + b + sigP + "\n"
    w.write(string1)

Answer 1

Question: Is there a way to speed this up or use better methodology?

尝试以下操作以查看这是否会提高性能：

Note: Didn't know the Values of col and max_column!
My Example uses 4 Columns and skips Column C.

Data:
['A1', 'B1', 'C1', 'D1'],
['A2', 'B2', 'C2', 'D2']

from openpyxl.utils import range_boundaries
min_col, min_row, max_col, max_row = range_boundaries('A1:D2')

for row_cells in ws.iter_rows(min_col=min_col, min_row=min_row,
                              max_col=max_col, max_row=max_row):

    # Slice Column Values up to B
    data = [cell.value for cell in row_cells[:2]]

    # Extend List with sliced Column Values from D up to End
    data.extend([cell.value for cell in row_cells[3:]])

    # Append db.get(Column A.value)
    data.append(db.get(row_cells[0].value, "notfound"))

    # Join all List Values delimited with \t
    print('{}'.format('\t'.join(data)))

    # Write to CSV
    #w.write(data)

Output:
A1 B1 D1 notfound
A2 B2 D2 notfound

测试 Python: 3.4.2 - openpyxl: 2.4.1

使用 openpyxl 处理非常大的文件 python

Handling very large files with openpyxl python

python

bigdata

openpyxl