如何在 for 循环中将数据帧添加到 Excel 工作簿？

Question

我有一个列表列表 (sub-lists)，其中每个 sub-list 都是一组字符。该列表来自数据的文本文件，其中连续的样本数据垂直对齐。这是一个示例：

""
"Test Method","Generic Stuff.msm"
"Sample I. D.","sed do eiusmod tempor incididunt ut labore et.mss"
"Specimen Number","1"

"Load (lbf)","Time (s)","Extension (in)"

48.081,3.150,0.000
77.307,3.200,0.000
98.159,3.250,0.000
53.256,3.300,0.000
42.476,3.350,0.000
67.080,3.400,0.000
17.786,3.450,0.000
82.600,3.500,0.001
50.644,3.550,0.001
97.122,3.600,0.001
/n
/n

所有示例都用双引号分隔，并以几行换行结尾。

Pandas 在允许我将这些字符转换为浮点数或将它们保留为字符串方面做得很好。所以，我决定通过 for-loop 放置子列表，调整数据实际开始的位置（每个样本都存在相同的 headers），并尝试将每个样本添加到它自己的 SHEET在单个 Excel 工作簿中。

代码如下：

source_file = r'input.txt'
base = os.path.splitext(source_file)[0]
excel_file = base + ".xlsx"

with open(excel_file, 'w') as fp:
    workbook = openpyxl.Workbook()
    initial_work_sheet = workbook.active
    initial_work_sheet.title = 'Create WorkBook'
    initial_work_sheet['A1'] = "Do With This Sheet As You Please"
    workbook.save(excel_file)

with open(source_file, 'r') as file:
    data = file.read().split('""')
    data = [i.split('\n') for i in data]
    data.remove([''])
    for i in np.arange(len(data)):
        data[i] = list(filter(None, data[i]))

source_WB = openpyxl.load_workbook(excel_file)

for sub_data in data:
    sub_data = [s.split(',') for s in sub_data][2:]
    df = pd.DataFrame(sub_data[2:], columns=sub_data[1])
    df['"Load (lbf)"'] = df['"Load (lbf)"'].astype(float)
    df['"Time (s)"'] = df['"Time (s)"'].astype(float)
    df['"Extension (in)"'] = df['"Extension (in)"'].astype(float)
    
    source_WB.create_sheet(' '.join(sub_data[0]))
    print("Writing ", ' '.join(sub_data[0]))

    if ' '.join(sub_data[0]) in source_WB.sheetnames:
        ws = source_WB[' '.join(sub_data[0])]
    else:
        ws = source_WB.active
    for r in dataframe_to_rows(df, index=False, header=True):
        ws.append(r)
    source_WB.save(excel_file)

最初，我试图在每次循环时加载整个工作簿，但事实证明这会消耗太多内存。然后我以 openpyxl 的优化模式结束，但即使那样它也会显着减慢。 “数据”列表大约有 30,000 多行，有 3 列。

使用openpyxl的优化模式，我希望将数据从“read_only”excel文件复制到“write_only”excel文件作为推荐在文档中。然后我将“write_only”excel 文件保存在“read_only”文件上，该文件充当下一个循环的源。

除了在第 10 个样本之后出现显着的减速之外，所有样本的所有数据都对齐在一个列中，除了最后一个样本，它按预期在 3 列中包含所有数据。

我已经尽我所能询问搜索引擎，但仍然找不到适合我正在做的事情。

我有很多数据，我需要将其放入 excel，如何快速完成？谢谢

Answer 1

我想通了！答案如下。我知道这可能到处都是，我根本不明白，但这能够非常快速地将我的数据帧写入 excel 工作簿，每个工作簿都有自己的 sheet。没有过多的内存消耗。

import os
import numpy as np
import pandas as pd
import openpyxl
from openpyxl.utils.dataframe import dataframe_to_rows

print("openpyxl version: ", openpyxl.__version__)

source_file = r'input.txt'

base = os.path.splitext(source_file)[0]
excel_file = base + ".xlsx"

with open(source_file, 'r') as file:
    data = file.read().split('""')
    data = [i.split('\n') for i in data]
    data.remove([''])
    for i in np.arange(len(data)):
        data[i] = list(filter(None, data[i]))

source_WB = openpyxl.Workbook(write_only=True)

for sub_data in data:
    sub_data = [s.split(',') for s in sub_data][2:]
    df = pd.DataFrame(sub_data[2:], columns=sub_data[1])
    df['"Load (lbf)"'] = df['"Load (lbf)"'].astype(float)
    df['"Time (s)"'] = df['"Time (s)"'].astype(float)
    df['"Extension (in)"'] = df['"Extension (in)"'].astype(float)

    source_WB.create_sheet(title=' '.join(sub_data[0]))
    print("Writing ", ' '.join(sub_data[0]))

    ws = source_WB[' '.join(sub_data[0])]
    for r in dataframe_to_rows(df, index=False, header=True):
        ws.append(r)

source_WB.save(excel_file)

Answer 2

我认为该方法可行，但 pandas 有一个内置的 excel 编写器函数，速度非常快。

%%timeit

with pd.ExcelWriter('path_to_file.xlsx', mode='w') as writer:
    df_1.to_excel(writer, sheet_name='Sheet1')
    df_2.to_excel(writer, sheet_name='Sheet2')
    df_3.to_excel(writer, sheet_name='Sheet3')
    df_4.to_excel(writer, sheet_name='Sheet4')
    df_5.to_excel(writer, sheet_name='Sheet5')

结果相当可观

需要注意的一件事是因为你提到它陷入困境，我用 5 个文件测试了上面的内容，每个文件 500 行和三列。

如何在 for 循环中将数据帧添加到 Excel 工作簿？

How to do add DataFrames to Excel workbook in a for loop?

python

excel

dataframe

openpyxl

raw-data