使用文件名行转换非结构化 CSV

Question

我正在使用一个输出 non-standard CSV 文件的系统。第 1 行始终包含文件名，后跟第 2 行中 table 的属性（有时包括逗号），第 3 行中的 table headers，然后是数量不等的数据行。在数据行之后，总是有两个空行并且模式重复（headers 在文件中总是相同的）。这是一个小例子：

Example Report
Geography:Boston, MA
Time,Product,Unit Sales
Week Ending 03-06-22,ITEM DESCRIPTION A,275
Week Ending 03-13-22,ITEM DESCRIPTION A,297
Week Ending 03-20-22,ITEM DESCRIPTION A,261


Example Report
Geography:New York, NY
Time,Product,Unit Sales
Week Ending 03-06-22,ITEM DESCRIPTION A,393
Week Ending 03-13-22,ITEM DESCRIPTION A,477
Week Ending 03-20-22,ITEM DESCRIPTION A,412


Example Report
Geography:Philadelphia, PA
Time,Product,Unit Sales
Week Ending 03-06-22,ITEM DESCRIPTION A,195
Week Ending 03-13-22,ITEM DESCRIPTION A,233
Week Ending 03-20-22,ITEM DESCRIPTION A,198

最终，我想丢弃文件名和额外的 header 行，并输出一个标准 CSV，该属性作为第一列。上面的例子应该是这样的：

Geography,Time,Product,Unit Sales
"Boston, MA",Week Ending 03-06-22,ITEM DESCRIPTION A,275
"Boston, MA",Week Ending 03-13-22,ITEM DESCRIPTION A,297
"Boston, MA",Week Ending 03-20-22,ITEM DESCRIPTION A,261
"New York, NY",Week Ending 03-06-22,ITEM DESCRIPTION A,393
"New York, NY",Week Ending 03-13-22,ITEM DESCRIPTION A,477
"New York, NY",Week Ending 03-20-22,ITEM DESCRIPTION A,412
"Philadelphia, PA",Week Ending 03-06-22,ITEM DESCRIPTION A,195
"Philadelphia, PA",Week Ending 03-13-22,ITEM DESCRIPTION A,233
"Philadelphia, PA",Week Ending 03-20-22,ITEM DESCRIPTION A,198

我习惯于在 python 中操作标准 CSV 文件，但是这个文件让我难以理解，其中混合了非结构化数据。

Answer 1

pandas.read_csv默认有一个参数skip_blank_lines=True。其他的我会在pandas.

处理

df1 = pd.read_csv('filename', skiprows=2, skipfooter=16)
df1[Geography']='Boston'

df2 = pd.read_csv('filename', skiprows=9, skipfooter=8)
df2[Geography']='Boston'

df3 = pd.read_csv('filename', skiprows=2, skipfooter=1) #adjust those, they might have errors
df3[Geography']='Boston'

df=pd.concat(df1,df2,df3)

我知道很难对多个数据执行此操作，但这是我能想到的最佳解决方案。祝您问题解决顺利！

Answer 2

迭代此类 CSV 的工作解决方案：

def read_strange_csv(filename):
    header_used = False
    with open(filename) as f:
        while True:
            line_filename = next(f).rstrip()
            line_attribute = next(f).rstrip()
            geography = line_attribute.split(':')[1]
            line_header = next(f).rstrip()
            if not header_used:
                yield f'Geography,{line_header}'
                header_used = True

            for line in f:
                line = line.rstrip()
                if not line:
                    break
                yield f'"{geography}",{line}'

            try:
                next(f)  # empty line
            except StopIteration:
                return


for row in read_strange_csv('example.csv'):
    print(row)

它打印出下面一行，如果需要，您可以直接将其保存到文件中：

Geography,Time,Product,Unit Sales
"Boston, MA",Week Ending 03-06-22,ITEM DESCRIPTION A,275
"Boston, MA",Week Ending 03-13-22,ITEM DESCRIPTION A,297
"Boston, MA",Week Ending 03-20-22,ITEM DESCRIPTION A,261
"New York, NY",Week Ending 03-06-22,ITEM DESCRIPTION A,393
"New York, NY",Week Ending 03-13-22,ITEM DESCRIPTION A,477
"New York, NY",Week Ending 03-20-22,ITEM DESCRIPTION A,412
"Philadelphia, PA",Week Ending 03-06-22,ITEM DESCRIPTION A,195
"Philadelphia, PA",Week Ending 03-13-22,ITEM DESCRIPTION A,233
"Philadelphia, PA",Week Ending 03-20-22,ITEM DESCRIPTION A,198

使用文件名行转换非结构化 CSV

Convert unstructured CSV with filename rows

python

csv