Python: json_normalize 一个 pandas 系列给出 TypeError
Python: json_normalize a pandas series gives TypeError
我在 pandas 系列 df["json"]
中有数万行像这样的 json 片段
[{
'IDs': [{
'lotId': '1',
'Id': '123456'
}],
'date': '2009-04-17',
'bidsCount': 2,
}, {
'IDs': [{
'lotId': '2',
'Id': '123456'
}],
'date': '2009-04-17',
'bidsCount': 4,
}, {
'IDs': [{
'lotId': '3',
'Id': '123456'
}],
'date': '2009-04-17',
'bidsCount': 8,
}]
原文件样本:
{"type": "OPEN","title": "rainbow","json": [{"IDs": [{"lotId": "1","Id": "123456"}],"date": "2009-04-17","bidsCount": 2,}, {"IDs": [{"lotId": "2","Id": "123456"}],"date": "2009-04-17","bidsCount": 4,}, {"IDs": [{"lotId": "3","Id": "123456"}],"date": "2009-04-17","bidsCount": 8,}]}
{"type": "CLOSED","title": "clouds","json": [{"IDs": [{"lotId": "1","Id": "23345"}],"date": "2009-05-17","bidsCount": 2,}, {"IDs": [{"lotId": "2","Id": "23345"}],"date": "2009-05-17","bidsCount": 4,}, {"IDs": [{"lotId": "3","Id": "23345"}],"date": "2009-05-17","bidsCount": 8,}]}
df = pd.read_json("file.json", lines=True)
我正在尝试将它们制作成数据框,类似于
Id lotId bidsCount date
123456 1 2 2009-04-17
123456 2 4 2009-04-17
123456 3 8 2009-04-17
通过使用
json_normalize(df["json"])
不过我得到
AttributeError: 'list' object has no attribute 'values'
我想 json 片段被视为一个列表,但我不知道如何让它工作。
感谢帮助!
我认为您的 df['json']
是一个嵌套列表。您可以使用 for 循环并连接数据帧以获得大数据帧,即
数据:
{"type": "OPEN","title": "rainbow","json": [{"IDs": [{"lotId": "1","Id": "123456"}],"date": "2009-04-17","bidsCount": 2,}, {"IDs": [{"lotId": "2","Id": "123456"}],"date": "2009-04-17","bidsCount": 4,}, {"IDs": [{"lotId": "3","Id": "123456"}],"date": "2009-04-17","bidsCount": 8,}]}
{"type": "CLOSED","title": "clouds","json": [{"IDs": [{"lotId": "1","Id": "23345"}],"date": "2009-05-17","bidsCount": 2,}, {"IDs": [{"lotId": "2","Id": "23345"}],"date": "2009-05-17","bidsCount": 4,}, {"IDs": [{"lotId": "3","Id": "23345"}],"date": "2009-05-17","bidsCount": 8,}]}
df = pd.read_json("file.json", lines=True)
数据帧:
new_df = pd.concat([pd.DataFrame(json_normalize(x)) for x in df['json']],ignore_index=True)
输出:
IDs bidsCount date
0 [{'Id': '123456', 'lotId': '1'}] 2 2009-04-17
1 [{'Id': '123456', 'lotId': '2'}] 4 2009-04-17
2 [{'Id': '123456', 'lotId': '3'}] 8 2009-04-17
3 [{'Id': '23345', 'lotId': '1'}] 2 2009-05-17
4 [{'Id': '23345', 'lotId': '2'}] 4 2009-05-17
5 [{'Id': '23345', 'lotId': '3'}] 8 2009-05-17
如果您希望 ID 的键作为列,那么您可以使用
new_df['lotId'] = [x[0]['lotId'] for x in new_df['IDs']]
new_df['IDs'] = [x[0]['Id'] for x in new_df['IDs']]
IDs bidsCount date lotId
0 123456 2 2009-04-17 1
1 123456 4 2009-04-17 2
2 123456 8 2009-04-17 3
3 23345 2 2009-05-17 1
4 23345 4 2009-05-17 2
5 23345 8 2009-05-17 3
我在 pandas 系列 df["json"]
[{
'IDs': [{
'lotId': '1',
'Id': '123456'
}],
'date': '2009-04-17',
'bidsCount': 2,
}, {
'IDs': [{
'lotId': '2',
'Id': '123456'
}],
'date': '2009-04-17',
'bidsCount': 4,
}, {
'IDs': [{
'lotId': '3',
'Id': '123456'
}],
'date': '2009-04-17',
'bidsCount': 8,
}]
原文件样本:
{"type": "OPEN","title": "rainbow","json": [{"IDs": [{"lotId": "1","Id": "123456"}],"date": "2009-04-17","bidsCount": 2,}, {"IDs": [{"lotId": "2","Id": "123456"}],"date": "2009-04-17","bidsCount": 4,}, {"IDs": [{"lotId": "3","Id": "123456"}],"date": "2009-04-17","bidsCount": 8,}]}
{"type": "CLOSED","title": "clouds","json": [{"IDs": [{"lotId": "1","Id": "23345"}],"date": "2009-05-17","bidsCount": 2,}, {"IDs": [{"lotId": "2","Id": "23345"}],"date": "2009-05-17","bidsCount": 4,}, {"IDs": [{"lotId": "3","Id": "23345"}],"date": "2009-05-17","bidsCount": 8,}]}
df = pd.read_json("file.json", lines=True)
我正在尝试将它们制作成数据框,类似于
Id lotId bidsCount date
123456 1 2 2009-04-17
123456 2 4 2009-04-17
123456 3 8 2009-04-17
通过使用
json_normalize(df["json"])
不过我得到
AttributeError: 'list' object has no attribute 'values'
我想 json 片段被视为一个列表,但我不知道如何让它工作。 感谢帮助!
我认为您的 df['json']
是一个嵌套列表。您可以使用 for 循环并连接数据帧以获得大数据帧,即
数据:
{"type": "OPEN","title": "rainbow","json": [{"IDs": [{"lotId": "1","Id": "123456"}],"date": "2009-04-17","bidsCount": 2,}, {"IDs": [{"lotId": "2","Id": "123456"}],"date": "2009-04-17","bidsCount": 4,}, {"IDs": [{"lotId": "3","Id": "123456"}],"date": "2009-04-17","bidsCount": 8,}]}
{"type": "CLOSED","title": "clouds","json": [{"IDs": [{"lotId": "1","Id": "23345"}],"date": "2009-05-17","bidsCount": 2,}, {"IDs": [{"lotId": "2","Id": "23345"}],"date": "2009-05-17","bidsCount": 4,}, {"IDs": [{"lotId": "3","Id": "23345"}],"date": "2009-05-17","bidsCount": 8,}]}
df = pd.read_json("file.json", lines=True)
数据帧:
new_df = pd.concat([pd.DataFrame(json_normalize(x)) for x in df['json']],ignore_index=True)
输出:
IDs bidsCount date 0 [{'Id': '123456', 'lotId': '1'}] 2 2009-04-17 1 [{'Id': '123456', 'lotId': '2'}] 4 2009-04-17 2 [{'Id': '123456', 'lotId': '3'}] 8 2009-04-17 3 [{'Id': '23345', 'lotId': '1'}] 2 2009-05-17 4 [{'Id': '23345', 'lotId': '2'}] 4 2009-05-17 5 [{'Id': '23345', 'lotId': '3'}] 8 2009-05-17
如果您希望 ID 的键作为列,那么您可以使用
new_df['lotId'] = [x[0]['lotId'] for x in new_df['IDs']]
new_df['IDs'] = [x[0]['Id'] for x in new_df['IDs']]
IDs bidsCount date lotId 0 123456 2 2009-04-17 1 1 123456 4 2009-04-17 2 2 123456 8 2009-04-17 3 3 23345 2 2009-05-17 1 4 23345 4 2009-05-17 2 5 23345 8 2009-05-17 3