由于数据类型不匹配 PySpark 无法解析列

Question

PySpark 中遇到错误：

pyspark.sql.utils.AnalysisException: "cannot resolve '`result_set`.`dates`.`trackers`['token']' due to data type mismatch: argument 2 requires integral type, however, ''token'' is of string type.;;\n'Project [result_parameters#517, result_set#518, <lambda>(result_set#518.dates.trackers[token]) AS result_set.dates.trackers.token#705]\n+- Relation[result_parameters#517,result_set#518] json\n"

数据结构：

-- result_set: struct (nullable = true)
 |    |-- currency: string (nullable = true)
 |    |-- dates: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- date: string (nullable = true)
 |    |    |    |-- trackers: array (nullable = true)
 |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |-- countries: array (nullable = true)
 |    |    |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |    |    |-- country: string (nullable = true)
 |    |    |    |    |    |    |    |-- os_names: array (nullable = true)
 |    |    |    |    |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |    |    |    |    |-- kpi_values: array (nullable = true)
 |    |    |    |    |    |    |    |    |    |    |-- element: double (containsNull = true)
 |    |    |    |    |    |    |    |    |    |-- os_name: string (nullable = true)
 |    |    |    |    |    |-- token: string (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- token: string (nullable = true)

我正在尝试创建一个视图来显示货币、日期和代币：

df.select('result_set.currency', 'result_set.dates.date', 'result_set.dates.trackers.token').show()

数据样本：

"result_set": {
        "token": "abcdef",
        "name": "Facebook",
        "currency": "EUR",
        "dates": [
            {
                "date": "2020-03-11",
                "trackers": [
                    {
                        "token": "12345",
                        "countries": [
                            {
                                "country": "am",
                                "os_names": [
                                    {
                                        "os_name": "android",
                                        "kpi_values": [
                                            0,
                                            0,
                                            0,
                                            0,
                                            0,
                                            0,
                                            1,
                                            0,
                                            0
                                        ]
                                    }
                                ]
                            },

我正在尝试根据 json 数据中的几个级别创建视图。

更新：

复制令牌

df.selectExpr('result_set.currency','explode(result_set.dates)').\
select("*","col.*").\
selectExpr("explode(trackers)","*").\
selectExpr("currency","date","explode(trackers)").\
select("currency","date","col.*").\
selectExpr("currency","date","token", "explode(countries)").\
select("currency","date","token", "col.*").\
selectExpr("currency","date","token", "country", "explode(os_names)").\
select("currency","date","token", "country", "col.*").\
selectExpr("currency","date","token", "country", "os_name", "explode(kpi_values)").\
show(20)

经过一些爆炸，现在令牌重复 8 次。

Answer 1

当您访问 array of structs 时，我们需要给出我们需要访问 array 中的哪个元素，即 0,1,2..等

如果我们需要 select 数组的所有元素，那么我们需要使用 explode().

Example:

df.printSchema()
#root
# |-- result_set: struct (nullable = true)
# |    |-- currency: string (nullable = true)
# |    |-- dates: array (nullable = true)
# |    |    |-- element: struct (containsNull = true)
# |    |    |    |-- date: string (nullable = true)
# |    |    |    |-- trackers: array (nullable = true)
# |    |    |    |    |-- element: struct (containsNull = true)
# |    |    |    |    |    |-- countries: array (nullable = true)
# |    |    |    |    |    |    |-- element: struct (containsNull = true)
# |    |    |    |    |    |    |    |-- country: string (nullable = true)
# |    |    |    |    |    |    |    |-- os_names: array (nullable = true)
# |    |    |    |    |    |    |    |    |-- element: struct (containsNull = true)
# |    |    |    |    |    |    |    |    |    |-- kpi_values: array (nullable = true)
# |    |    |    |    |    |    |    |    |    |    |-- element: long (containsNull = true)
# |    |    |    |    |    |    |    |    |    |-- os_name: string (nullable = true)
# |    |    |    |    |    |-- token: string (nullable = true)
# |    |-- name: string (nullable = true)
# |    |-- token: string (nullable = true)

#accessing token,date from array
df.selectExpr('result_set.dates.trackers[0].token','result_set.currency', 'result_set.dates.date').show()
#+--------------------------------------------------+--------+------------+
#|result_set.dates.trackers AS trackers#194[0].token|currency|        date|
#+--------------------------------------------------+--------+------------+
#|                                           [12345]|     EUR|[2020-03-11]|
#+--------------------------------------------------+--------+------------+

#accessing first elements from dates, trackers array and extracting date,token values
df.selectExpr('result_set.dates[0].trackers[0].token as token','result_set.currency', 'result_set.dates[0].date as date').show()
#+-----+--------+----------+
#|token|currency|      date|
#+-----+--------+----------+
#|12345|     EUR|2020-03-11|
#+-----+--------+----------+

#if you need to select all elements of array then we need to explode the array and select the data
df.selectExpr('result_set.currency','explode(result_set.dates)').\
select("*","col.*").\
selectExpr("explode(trackers)","*").\
selectExpr("currency","date","explode(trackers)").\
select("currency","date","col.*").\
select("currency","date","token").\
show()

#+--------+----------+-----+
#|currency|      date|token|
#+--------+----------+-----+
#|     EUR|2020-03-11|12345|
#+--------+----------+-----+

由于数据类型不匹配 PySpark 无法解析列

cannot resolve column due to data type mismatch PySpark

python

pyspark

pyspark-dataframes