Spark 无法读取 CSV 文件并转换为数据集
Spark can't read CSV file and convert to Dataset
当我使用 Spark 读取 CSV 文件并将其转换为数据集时，出现以下错误。我想不通原因。我的代码在下面提供。还 http://eforexcel.com/wp/wp-content/uploads/2017/07/10000-Sales-Records.zip 下载 CSV 文件。
我正在使用 Scala：2.12.3，Spark：2.4.0。
错误信息：
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`itemType`' given input columns: [Order ID, Total Profit, Country, Total Revenue, Ship Date, Unit Cost, Sales Channel, Unit Price, Total Cost, Units Sold, Order Date, Order Priority, Region, Item Type];
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis.applyOrElse(CheckAnalysis.scala:110)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis.applyOrElse(CheckAnalysis.scala:107)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp(TreeNode.scala:278)
...
...
这是我的代码：
import spark.implicits._
case class Sales(region: String,
country: String,
itemType: String,
salesChannel: String,
orderPriority: String,
orderDate: String,
orderId: Long,
shipDate: String,
unitsSold: Integer,
unitsPrice: Double,
unitCost: Double,
totalRevenue: Double,
totalCost: Double,
totalProfit: Double
)
val ds = spark.read
.option("header", "true")
.option("inferSchema", "true")
.csv("src/main/resources/datasets/10000 Sales Records.csv")
.as[Sales]
可能在输入文件中包含 header - 地区、国家、项目类型、销售渠道、订单优先级、订单日期、订单 ID、发货日期、单位已售出、单价、单位成本、总收入、总成本、总利润
在输入文件或案例中编辑 header class
名称：项目类型（带有 space），如果 class 没有 space
csv header 列与大小写 class 不匹配。
从 csv header，您需要修改数据以使其与您的案例相匹配 class。那就是你需要删除空格，将第二个单词大写。以下解决方法适合您。
请注意，我将 unitsPrice: Double, 更改为你的 unitPrice class。
val ds = spark.read
.option("header", "true")
.option("inferSchema", "true")
.csv("in/10000_Sales_Records.csv")
ds.printSchema()
val sch1 = ds.columns.map( x=> x match { case a if a.contains(" ") => { val q=a.split(" ");q(0)+q(1).capitalize } case a => a.toLowerCase } )
val ds2 = ds.toDF(sch1:_*)
ds2.printSchema()
val ds3 = ds2.as[Sales]
ds3.show(false)
结果：
root
|-- Region: string (nullable = true)
|-- Country: string (nullable = true)
|-- Item Type: string (nullable = true)
|-- Sales Channel: string (nullable = true)
|-- Order Priority: string (nullable = true)
|-- Order Date: string (nullable = true)
|-- Order ID: integer (nullable = true)
|-- Ship Date: string (nullable = true)
|-- Units Sold: integer (nullable = true)
|-- Unit Price: double (nullable = true)
|-- Unit Cost: double (nullable = true)
|-- Total Revenue: double (nullable = true)
|-- Total Cost: double (nullable = true)
|-- Total Profit: double (nullable = true)
root
|-- region: string (nullable = true)
|-- country: string (nullable = true)
|-- ItemType: string (nullable = true)
|-- SalesChannel: string (nullable = true)
|-- OrderPriority: string (nullable = true)
|-- OrderDate: string (nullable = true)
|-- OrderID: integer (nullable = true)
|-- ShipDate: string (nullable = true)
|-- UnitsSold: integer (nullable = true)
|-- UnitPrice: double (nullable = true)
|-- UnitCost: double (nullable = true)
|-- TotalRevenue: double (nullable = true)
|-- TotalCost: double (nullable = true)
|-- TotalProfit: double (nullable = true)
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
|region |country |ItemType |SalesChannel|OrderPriority|OrderDate |OrderID |ShipDate |UnitsSold|UnitPrice|UnitCost|TotalRevenue|TotalCost |TotalProfit|
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
|Sub-Saharan Africa |Chad |Office Supplies|Online |L |1/27/2011 |292494523|2/12/2011 |4484 |651.21 |524.96 |2920025.64 |2353920.64|566105.0 |
|Europe |Latvia |Beverages |Online |C |12/28/2015|361825549|1/23/2016 |1075 |47.45 |31.79 |51008.75 |34174.25 |16834.5 |
|Middle East and North Africa |Pakistan |Vegetables |Offline |C |1/13/2011 |141515767|2/1/2011 |6515 |154.06 |90.93 |1003700.9 |592408.95 |411291.95 |
|Sub-Saharan Africa |Democratic Republic of the Congo|Household |Online |C |9/11/2012 |500364005|10/6/2012 |7683 |668.27 |502.54 |5134318.41 |3861014.82|1273303.59 |
|Europe |Czech Republic |Beverages |Online |C |10/27/2015|127481591|12/5/2015 |3491 |47.45 |31.79 |165647.95 |110978.89 |54669.06 |
|Sub-Saharan Africa |South Africa |Beverages |Offline |H |7/10/2012 |482292354|8/21/2012 |9880 |47.45 |31.79 |468806.0 |314085.2 |154720.8 |
|Asia |Laos |Vegetables |Online |L |2/20/2011 |844532620|3/20/2011 |4825 |154.06 |90.93 |743339.5 |438737.25 |304602.25 |
|Asia |China |Baby Food |Online |C |4/10/2017 |564251220|5/12/2017 |3330 |255.28 |159.42 |850082.4 |530868.6 |319213.8 |
|Sub-Saharan Africa |Eritrea |Meat |Online |L |11/21/2014|411809480|1/10/2015 |2431 |421.89 |364.69 |1025614.59 |886561.39 |139053.2 |
|Central America and the Caribbean|Haiti |Office Supplies|Online |C |7/4/2015 |327881228|7/20/2015 |6197 |651.21 |524.96 |4035548.37 |3253177.12|782371.25 |
|Sub-Saharan Africa |Zambia |Cereal |Offline |M |7/26/2016 |773452794|8/24/2016 |724 |205.7 |117.11 |148926.8 |84787.64 |64139.16 |
|Europe |Bosnia and Herzegovina |Baby Food |Offline |M |10/20/2012|479823005|11/15/2012|9145 |255.28 |159.42 |2334535.6 |1457895.9 |876639.7 |
|Europe |Germany |Office Supplies|Online |C |2/22/2015 |498603188|2/27/2015 |6618 |651.21 |524.96 |4309707.78 |3474185.28|835522.5 |
|Asia |India |Household |Online |C |8/27/2016 |151717174|9/2/2016 |5338 |668.27 |502.54 |3567225.26 |2682558.52|884666.74 |
|Middle East and North Africa |Algeria |Clothes |Offline |C |6/21/2011 |181401288|7/21/2011 |9527 |109.28 |35.84 |1041110.56 |341447.68 |699662.88 |
|Australia and Oceania |Palau |Snacks |Offline |L |9/19/2013 |500204360|10/4/2013 |441 |152.58 |97.44 |67287.78 |42971.04 |24316.74 |
|Central America and the Caribbean|Cuba |Beverages |Online |H |11/15/2015|640987718|11/30/2015|1365 |47.45 |31.79 |64769.25 |43393.35 |21375.9 |
|Europe |Vatican City |Beverages |Online |L |4/6/2015 |206925189|4/27/2015 |2617 |47.45 |31.79 |124176.65 |83194.43 |40982.22 |
|Middle East and North Africa |Lebanon |Personal Care |Offline |H |4/12/2010 |221503102|5/19/2010 |6545 |81.73 |56.67 |534922.85 |370905.15 |164017.7 |
|Europe |Lithuania |Snacks |Offline |H |9/26/2011 |878520286|10/2/2011 |2530 |152.58 |97.44 |386027.4 |246523.2 |139504.2 |
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
only showing top 20 rows
当我使用 Spark 读取 CSV 文件并将其转换为数据集时，出现以下错误。我想不通原因。我的代码在下面提供。还 http://eforexcel.com/wp/wp-content/uploads/2017/07/10000-Sales-Records.zip 下载 CSV 文件。
我正在使用 Scala：2.12.3，Spark：2.4.0。
错误信息：
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`itemType`' given input columns: [Order ID, Total Profit, Country, Total Revenue, Ship Date, Unit Cost, Sales Channel, Unit Price, Total Cost, Units Sold, Order Date, Order Priority, Region, Item Type];
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis.applyOrElse(CheckAnalysis.scala:110)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis.applyOrElse(CheckAnalysis.scala:107)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp(TreeNode.scala:278)
...
...
这是我的代码：
import spark.implicits._
case class Sales(region: String,
country: String,
itemType: String,
salesChannel: String,
orderPriority: String,
orderDate: String,
orderId: Long,
shipDate: String,
unitsSold: Integer,
unitsPrice: Double,
unitCost: Double,
totalRevenue: Double,
totalCost: Double,
totalProfit: Double
)
val ds = spark.read
.option("header", "true")
.option("inferSchema", "true")
.csv("src/main/resources/datasets/10000 Sales Records.csv")
.as[Sales]
可能在输入文件中包含 header - 地区、国家、项目类型、销售渠道、订单优先级、订单日期、订单 ID、发货日期、单位已售出、单价、单位成本、总收入、总成本、总利润
在输入文件或案例中编辑 header class
名称：项目类型（带有 space），如果 class 没有 space
csv header 列与大小写 class 不匹配。 从 csv header，您需要修改数据以使其与您的案例相匹配 class。那就是你需要删除空格，将第二个单词大写。以下解决方法适合您。
请注意，我将 unitsPrice: Double, 更改为你的 unitPrice class。
val ds = spark.read
.option("header", "true")
.option("inferSchema", "true")
.csv("in/10000_Sales_Records.csv")
ds.printSchema()
val sch1 = ds.columns.map( x=> x match { case a if a.contains(" ") => { val q=a.split(" ");q(0)+q(1).capitalize } case a => a.toLowerCase } )
val ds2 = ds.toDF(sch1:_*)
ds2.printSchema()
val ds3 = ds2.as[Sales]
ds3.show(false)
结果：
root
|-- Region: string (nullable = true)
|-- Country: string (nullable = true)
|-- Item Type: string (nullable = true)
|-- Sales Channel: string (nullable = true)
|-- Order Priority: string (nullable = true)
|-- Order Date: string (nullable = true)
|-- Order ID: integer (nullable = true)
|-- Ship Date: string (nullable = true)
|-- Units Sold: integer (nullable = true)
|-- Unit Price: double (nullable = true)
|-- Unit Cost: double (nullable = true)
|-- Total Revenue: double (nullable = true)
|-- Total Cost: double (nullable = true)
|-- Total Profit: double (nullable = true)
root
|-- region: string (nullable = true)
|-- country: string (nullable = true)
|-- ItemType: string (nullable = true)
|-- SalesChannel: string (nullable = true)
|-- OrderPriority: string (nullable = true)
|-- OrderDate: string (nullable = true)
|-- OrderID: integer (nullable = true)
|-- ShipDate: string (nullable = true)
|-- UnitsSold: integer (nullable = true)
|-- UnitPrice: double (nullable = true)
|-- UnitCost: double (nullable = true)
|-- TotalRevenue: double (nullable = true)
|-- TotalCost: double (nullable = true)
|-- TotalProfit: double (nullable = true)
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
|region |country |ItemType |SalesChannel|OrderPriority|OrderDate |OrderID |ShipDate |UnitsSold|UnitPrice|UnitCost|TotalRevenue|TotalCost |TotalProfit|
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
|Sub-Saharan Africa |Chad |Office Supplies|Online |L |1/27/2011 |292494523|2/12/2011 |4484 |651.21 |524.96 |2920025.64 |2353920.64|566105.0 |
|Europe |Latvia |Beverages |Online |C |12/28/2015|361825549|1/23/2016 |1075 |47.45 |31.79 |51008.75 |34174.25 |16834.5 |
|Middle East and North Africa |Pakistan |Vegetables |Offline |C |1/13/2011 |141515767|2/1/2011 |6515 |154.06 |90.93 |1003700.9 |592408.95 |411291.95 |
|Sub-Saharan Africa |Democratic Republic of the Congo|Household |Online |C |9/11/2012 |500364005|10/6/2012 |7683 |668.27 |502.54 |5134318.41 |3861014.82|1273303.59 |
|Europe |Czech Republic |Beverages |Online |C |10/27/2015|127481591|12/5/2015 |3491 |47.45 |31.79 |165647.95 |110978.89 |54669.06 |
|Sub-Saharan Africa |South Africa |Beverages |Offline |H |7/10/2012 |482292354|8/21/2012 |9880 |47.45 |31.79 |468806.0 |314085.2 |154720.8 |
|Asia |Laos |Vegetables |Online |L |2/20/2011 |844532620|3/20/2011 |4825 |154.06 |90.93 |743339.5 |438737.25 |304602.25 |
|Asia |China |Baby Food |Online |C |4/10/2017 |564251220|5/12/2017 |3330 |255.28 |159.42 |850082.4 |530868.6 |319213.8 |
|Sub-Saharan Africa |Eritrea |Meat |Online |L |11/21/2014|411809480|1/10/2015 |2431 |421.89 |364.69 |1025614.59 |886561.39 |139053.2 |
|Central America and the Caribbean|Haiti |Office Supplies|Online |C |7/4/2015 |327881228|7/20/2015 |6197 |651.21 |524.96 |4035548.37 |3253177.12|782371.25 |
|Sub-Saharan Africa |Zambia |Cereal |Offline |M |7/26/2016 |773452794|8/24/2016 |724 |205.7 |117.11 |148926.8 |84787.64 |64139.16 |
|Europe |Bosnia and Herzegovina |Baby Food |Offline |M |10/20/2012|479823005|11/15/2012|9145 |255.28 |159.42 |2334535.6 |1457895.9 |876639.7 |
|Europe |Germany |Office Supplies|Online |C |2/22/2015 |498603188|2/27/2015 |6618 |651.21 |524.96 |4309707.78 |3474185.28|835522.5 |
|Asia |India |Household |Online |C |8/27/2016 |151717174|9/2/2016 |5338 |668.27 |502.54 |3567225.26 |2682558.52|884666.74 |
|Middle East and North Africa |Algeria |Clothes |Offline |C |6/21/2011 |181401288|7/21/2011 |9527 |109.28 |35.84 |1041110.56 |341447.68 |699662.88 |
|Australia and Oceania |Palau |Snacks |Offline |L |9/19/2013 |500204360|10/4/2013 |441 |152.58 |97.44 |67287.78 |42971.04 |24316.74 |
|Central America and the Caribbean|Cuba |Beverages |Online |H |11/15/2015|640987718|11/30/2015|1365 |47.45 |31.79 |64769.25 |43393.35 |21375.9 |
|Europe |Vatican City |Beverages |Online |L |4/6/2015 |206925189|4/27/2015 |2617 |47.45 |31.79 |124176.65 |83194.43 |40982.22 |
|Middle East and North Africa |Lebanon |Personal Care |Offline |H |4/12/2010 |221503102|5/19/2010 |6545 |81.73 |56.67 |534922.85 |370905.15 |164017.7 |
|Europe |Lithuania |Snacks |Offline |H |9/26/2011 |878520286|10/2/2011 |2530 |152.58 |97.44 |386027.4 |246523.2 |139504.2 |
+---------------------------------+--------------------------------+---------------+------------+-------------+----------+---------+----------+---------+---------+--------+------------+----------+-----------+
only showing top 20 rows