如何导入没有分隔符且逗号代表缺失值的csv或txt文件

Question

我正在尝试将 CSV 中的值导入 10 列：有些有数字，有些有逗号，但逗号缺少值，因此没有分隔符：

2000-01-05,,-0.8803936956661669,,,,,,,-0.8316023477879247,

2000-01-06,,,,,,,,,,

2000-01-07,,,,,,,,,-0.3133976053851764,

2000-01-10,-0.26878027549229977,,,,,,,,,

2000-01-11,,,,,,,,1.0787295663966179,,

我尝试了下面的代码，但它删除了左侧的日期列：

data = np.genfromtxt('Book7.txt', invalid_raise = True, usemask = False)
datanew = data[:,~np.all(np.isnan(data), axis = 0)]

Answer 1

我不知道您希望缺失数据是什么，但是此代码将日期列转换为 datetime.date，同时将缺失值设置为 NaN。

import numpy as np
import datetime

def convert_iso_string_to_date(s):
    year, month, day = (int(x) for x in s.decode("ascii").split("-"))
    return datetime.date(year, month, day)

data = np.genfromtxt("test.txt", delimiter=",", converters={0: convert_iso_string_to_date}, invalid_raise=True, usemask=False)
print(data)

[(datetime.date(2000, 1, 5),         nan, -0.8803937, nan, nan, nan, nan, nan,        nan, -0.83160235, nan)
 (datetime.date(2000, 1, 6),         nan,        nan, nan, nan, nan, nan, nan,        nan,         nan, nan)
 (datetime.date(2000, 1, 7),         nan,        nan, nan, nan, nan, nan, nan,        nan, -0.31339761, nan)
 (datetime.date(2000, 1, 10), -0.26878028,        nan, nan, nan, nan, nan, nan,        nan,         nan, nan)
 (datetime.date(2000, 1, 11),         nan,        nan, nan, nan, nan, nan, nan, 1.07872957,         nan, nan)]

Answer 2

不确定 numpy 是首选还是强制性的。 pandas 无需额外代码即可完成此操作：

import io
import pandas as pd

text = """2000-01-05,,-0.8803936956661669,,,,,,,-0.8316023477879247,

2000-01-06,,,,,,,,,,

2000-01-07,,,,,,,,,-0.3133976053851764,

2000-01-10,-0.26878027549229977,,,,,,,,,

2000-01-11,,,,,,,,1.0787295663966179,,"""

csv = io.StringIO(text)

df = pd.DataFrame([cell.split(',') for cell in csv])

print(df)

输出：

           0                     1   ...                   9     10
0  2000-01-05                        ...  -0.8316023477879247    \n
1          \n                  None  ...                 None  None
2  2000-01-06                        ...                         \n
3          \n                  None  ...                 None  None
4  2000-01-07                        ...  -0.3133976053851764    \n
5          \n                  None  ...                 None  None
6  2000-01-10  -0.26878027549229977  ...                         \n
7          \n                  None  ...                 None  None
8  2000-01-11                        ...                           

[9 rows x 11 columns]

虽然您可能想要删除空行。

Answer 3

您可以简单地使用 python 内置函数：

from numpy import array

with open('Book7.txt') as file:
    data = file.readlines()

matrix = []
for line in data:
    if line != '\n':
        matrix.append(line.split(',')[0:10])
matrix = array(matrix)

如何导入没有分隔符且逗号代表缺失值的csv或txt文件

How to import csv or txt file where there is no delimiter and commas represent missing values

python

csv

numpy

delimiter