Pandas 半结构化 JSON 数据框到简单 Pandas 数据框
Pandas semi structured JSON data frame to simple Pandas dataframe
我有一个从 redshift 集群中获取的数据块。
前 4 列由“|”分隔那么 2 列是 JSON.
XXX|ABANDONED|1197|11|"{""currency"":""EUR"" item_id"":""143"" type"":""FLIGHT"" name"":""PAR-FEZ"" price"":1111 origin"":""PAR"" destination"":""FEZ"" merchant"":""GOV"" flight_type"":""OW"" flight_segment"":[{ origin"":""ORY"" destination"":""FEZ"" departure_date_time"":""2015-08-02T07:20"" arrival_date_time"":""2015-08-02T09:05"" carrier"":""AT"" f_class"":""ECONOMY""}]}"|"{""type"":""FLIGHT"" name"":""FI_ORY-OUD"" item_id"":""FLIGHT"" currency"":""EUR"" price"":111 origin"":""ORY"" destination"":""OUD"" flight_type"":""OW"" flight_segment"":[{""origin"":""ORY"" destination"":""OUD"" departure_date_time"":""2015-08-02T13:55"" arrival_date_time"":""2015-08-02T15:30"" flight_number"":""AT625"" carrier"":""AT"" f_class"":""ECONOMIC_DISCOUNTED""}]}"
在 Python 2.7 中工作想分离出 JSON 值并将其转换为 Pandas 数据帧，但我在 pyparsing 方面缺乏经验。
我的方法是将文件作为 Pandas 数据框读入，其中包含“|”作为分隔符，而不是获取包含 JSON 的列并使用 'JSON_normalise' 将其展平，但 JSON_normalise 不会索引熊猫的列
我已经找到了解决方案 here and here 但一个不适合我的 'mixed data' 另一个是对于相当大的 JSON 文件
有关如何在此数据上部署 Pyparsing 的任何提示都将非常有帮助。
谢谢
Pyparsing: Parsing semi-JSON nested plaintext data to a list
Parsing semi-structured json data(Python/R)
将您上面的输入字符串作为一个名为 'data' 的变量，这个 Python+pyparsing 代码将对它有所了解。不幸的是，第四个'|'右边的东西不是真的 JSON。幸运的是，它 格式足够好，可以轻松解析它。请参阅下面程序中的嵌入式注释：
from pyparsing import *
from datetime import datetime
# for the most part, we suppress punctuation - it's important at parse time
# but just gets in the way afterwards
LBRACE,RBRACE,COLON,DBLQ,LBRACK,RBRACK = map(Suppress, '{}:"[]')
DBLQ2 = DBLQ + DBLQ
# define some scalar value expressions, including parse-time conversion parse actions
realnum = Regex(r'[+-]?\d+\.\d*').setParseAction(lambda t:float(t[0]))
integer = Regex(r'[+-]?\d+').setParseAction(lambda t:int(t[0]))
timestamp = Regex(r'""\d{4}-\d{2}-\d{2}T\d{2}:\d{2}""')
timestamp.setParseAction(lambda t: datetime.strptime(t[0][2:-2],'%Y-%m-%dT%H:%M'))
string_value = QuotedString('""')
# define our base key ':' value expression; use a Forward() placeholder
# for now for value, since these things can be recursive
key = Optional(DBLQ2) + Word(alphas, alphanums+'_') + DBLQ2
value = Forward()
key_value = Group(key + COLON + value)
# objects can be values too - use the Dict class to capture keys as field names
obj = Group(Dict(LBRACE + OneOrMore(key_value) + RBRACE))
objlist = (LBRACK + ZeroOrMore(obj) + RBRACK)
# define expression for previously-declared value, using <<= operator
value <<= timestamp | string_value | realnum | integer | obj | Group(objlist)
# the outermost objects are enclosed in "s, and list of them can be given with '|' delims
quotedObj = DBLQ + obj + DBLQ
obsList = delimitedList(quotedObj, delim='|')
现在将该解析器应用于您的 'data'：
fields = data.split('|',4)
result = obsList.parseString(fields[-1])
# we get back a list of objects, dump them out
for r in result:
print r.dump()
print
给出：
[['currency', 'EUR'], ['item_id', '143'], ['type', 'FLIGHT'], ['name', 'PAR-FEZ'], ['price', 1111], ['origin', 'PAR'], ['destination', 'FEZ'], ['merchant', 'GOV'], ['flight_type', 'OW'], ['flight_segment', [[['origin', 'ORY'], ['destination', 'FEZ'], ['departure_date_time', datetime.datetime(2015, 8, 2, 7, 20)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 9, 5)], ['carrier', 'AT'], ['f_class', 'ECONOMY']]]]]
- currency: EUR
- destination: FEZ
- flight_segment:
[0]:
[['origin', 'ORY'], ['destination', 'FEZ'], ['departure_date_time', datetime.datetime(2015, 8, 2, 7, 20)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 9, 5)], ['carrier', 'AT'], ['f_class', 'ECONOMY']]
- arrival_date_time: 2015-08-02 09:05:00
- carrier: AT
- departure_date_time: 2015-08-02 07:20:00
- destination: FEZ
- f_class: ECONOMY
- origin: ORY
- flight_type: OW
- item_id: 143
- merchant: GOV
- name: PAR-FEZ
- origin: PAR
- price: 1111
- type: FLIGHT
[['type', 'FLIGHT'], ['name', 'FI_ORY-OUD'], ['item_id', 'FLIGHT'], ['currency', 'EUR'], ['price', 111], ['origin', 'ORY'], ['destination', 'OUD'], ['flight_type', 'OW'], ['flight_segment', [[['origin', 'ORY'], ['destination', 'OUD'], ['departure_date_time', datetime.datetime(2015, 8, 2, 13, 55)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 15, 30)], ['flight_number', 'AT625'], ['carrier', 'AT'], ['f_class', 'ECONOMIC_DISCOUNTED']]]]]
- currency: EUR
- destination: OUD
- flight_segment:
[0]:
[['origin', 'ORY'], ['destination', 'OUD'], ['departure_date_time', datetime.datetime(2015, 8, 2, 13, 55)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 15, 30)], ['flight_number', 'AT625'], ['carrier', 'AT'], ['f_class', 'ECONOMIC_DISCOUNTED']]
- arrival_date_time: 2015-08-02 15:30:00
- carrier: AT
- departure_date_time: 2015-08-02 13:55:00
- destination: OUD
- f_class: ECONOMIC_DISCOUNTED
- flight_number: AT625
- origin: ORY
- flight_type: OW
- item_id: FLIGHT
- name: FI_ORY-OUD
- origin: ORY
- price: 111
- type: FLIGHT
请注意，不是字符串的值（整数、时间戳等）已被转换为 Python 类型。由于字段名称保存为字典键，您可以按名称访问字段，如：
res[0].currency
res[0].price
res[0].destination
res[0].flight_segment[0].origin
len(res[0].flight_segment) # gives how many segments
我有一个从 redshift 集群中获取的数据块。 前 4 列由“|”分隔那么 2 列是 JSON.
XXX|ABANDONED|1197|11|"{""currency"":""EUR"" item_id"":""143"" type"":""FLIGHT"" name"":""PAR-FEZ"" price"":1111 origin"":""PAR"" destination"":""FEZ"" merchant"":""GOV"" flight_type"":""OW"" flight_segment"":[{ origin"":""ORY"" destination"":""FEZ"" departure_date_time"":""2015-08-02T07:20"" arrival_date_time"":""2015-08-02T09:05"" carrier"":""AT"" f_class"":""ECONOMY""}]}"|"{""type"":""FLIGHT"" name"":""FI_ORY-OUD"" item_id"":""FLIGHT"" currency"":""EUR"" price"":111 origin"":""ORY"" destination"":""OUD"" flight_type"":""OW"" flight_segment"":[{""origin"":""ORY"" destination"":""OUD"" departure_date_time"":""2015-08-02T13:55"" arrival_date_time"":""2015-08-02T15:30"" flight_number"":""AT625"" carrier"":""AT"" f_class"":""ECONOMIC_DISCOUNTED""}]}"
在 Python 2.7 中工作想分离出 JSON 值并将其转换为 Pandas 数据帧，但我在 pyparsing 方面缺乏经验。
我的方法是将文件作为 Pandas 数据框读入，其中包含“|”作为分隔符，而不是获取包含 JSON 的列并使用 'JSON_normalise' 将其展平，但 JSON_normalise 不会索引熊猫的列
我已经找到了解决方案 here and here 但一个不适合我的 'mixed data' 另一个是对于相当大的 JSON 文件
有关如何在此数据上部署 Pyparsing 的任何提示都将非常有帮助。 谢谢
Pyparsing: Parsing semi-JSON nested plaintext data to a list
Parsing semi-structured json data(Python/R)
将您上面的输入字符串作为一个名为 'data' 的变量，这个 Python+pyparsing 代码将对它有所了解。不幸的是，第四个'|'右边的东西不是真的 JSON。幸运的是，它 格式足够好，可以轻松解析它。请参阅下面程序中的嵌入式注释：
from pyparsing import *
from datetime import datetime
# for the most part, we suppress punctuation - it's important at parse time
# but just gets in the way afterwards
LBRACE,RBRACE,COLON,DBLQ,LBRACK,RBRACK = map(Suppress, '{}:"[]')
DBLQ2 = DBLQ + DBLQ
# define some scalar value expressions, including parse-time conversion parse actions
realnum = Regex(r'[+-]?\d+\.\d*').setParseAction(lambda t:float(t[0]))
integer = Regex(r'[+-]?\d+').setParseAction(lambda t:int(t[0]))
timestamp = Regex(r'""\d{4}-\d{2}-\d{2}T\d{2}:\d{2}""')
timestamp.setParseAction(lambda t: datetime.strptime(t[0][2:-2],'%Y-%m-%dT%H:%M'))
string_value = QuotedString('""')
# define our base key ':' value expression; use a Forward() placeholder
# for now for value, since these things can be recursive
key = Optional(DBLQ2) + Word(alphas, alphanums+'_') + DBLQ2
value = Forward()
key_value = Group(key + COLON + value)
# objects can be values too - use the Dict class to capture keys as field names
obj = Group(Dict(LBRACE + OneOrMore(key_value) + RBRACE))
objlist = (LBRACK + ZeroOrMore(obj) + RBRACK)
# define expression for previously-declared value, using <<= operator
value <<= timestamp | string_value | realnum | integer | obj | Group(objlist)
# the outermost objects are enclosed in "s, and list of them can be given with '|' delims
quotedObj = DBLQ + obj + DBLQ
obsList = delimitedList(quotedObj, delim='|')
现在将该解析器应用于您的 'data'：
fields = data.split('|',4)
result = obsList.parseString(fields[-1])
# we get back a list of objects, dump them out
for r in result:
print r.dump()
print
给出：
[['currency', 'EUR'], ['item_id', '143'], ['type', 'FLIGHT'], ['name', 'PAR-FEZ'], ['price', 1111], ['origin', 'PAR'], ['destination', 'FEZ'], ['merchant', 'GOV'], ['flight_type', 'OW'], ['flight_segment', [[['origin', 'ORY'], ['destination', 'FEZ'], ['departure_date_time', datetime.datetime(2015, 8, 2, 7, 20)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 9, 5)], ['carrier', 'AT'], ['f_class', 'ECONOMY']]]]]
- currency: EUR
- destination: FEZ
- flight_segment:
[0]:
[['origin', 'ORY'], ['destination', 'FEZ'], ['departure_date_time', datetime.datetime(2015, 8, 2, 7, 20)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 9, 5)], ['carrier', 'AT'], ['f_class', 'ECONOMY']]
- arrival_date_time: 2015-08-02 09:05:00
- carrier: AT
- departure_date_time: 2015-08-02 07:20:00
- destination: FEZ
- f_class: ECONOMY
- origin: ORY
- flight_type: OW
- item_id: 143
- merchant: GOV
- name: PAR-FEZ
- origin: PAR
- price: 1111
- type: FLIGHT
[['type', 'FLIGHT'], ['name', 'FI_ORY-OUD'], ['item_id', 'FLIGHT'], ['currency', 'EUR'], ['price', 111], ['origin', 'ORY'], ['destination', 'OUD'], ['flight_type', 'OW'], ['flight_segment', [[['origin', 'ORY'], ['destination', 'OUD'], ['departure_date_time', datetime.datetime(2015, 8, 2, 13, 55)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 15, 30)], ['flight_number', 'AT625'], ['carrier', 'AT'], ['f_class', 'ECONOMIC_DISCOUNTED']]]]]
- currency: EUR
- destination: OUD
- flight_segment:
[0]:
[['origin', 'ORY'], ['destination', 'OUD'], ['departure_date_time', datetime.datetime(2015, 8, 2, 13, 55)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 15, 30)], ['flight_number', 'AT625'], ['carrier', 'AT'], ['f_class', 'ECONOMIC_DISCOUNTED']]
- arrival_date_time: 2015-08-02 15:30:00
- carrier: AT
- departure_date_time: 2015-08-02 13:55:00
- destination: OUD
- f_class: ECONOMIC_DISCOUNTED
- flight_number: AT625
- origin: ORY
- flight_type: OW
- item_id: FLIGHT
- name: FI_ORY-OUD
- origin: ORY
- price: 111
- type: FLIGHT
请注意，不是字符串的值（整数、时间戳等）已被转换为 Python 类型。由于字段名称保存为字典键，您可以按名称访问字段，如：
res[0].currency
res[0].price
res[0].destination
res[0].flight_segment[0].origin
len(res[0].flight_segment) # gives how many segments