TypeError: field Customer: Can not merge type <class 'pyspark.sql.types.StringType'> and <class 'pyspark.sql.types.DoubleType'>

Question

SL No:  Customer    Month   Amount
1   A1  12-Jan-04   495414.75
2   A1  3-Jan-04    245899.02
3   A1  15-Jan-04   259490.06

我的Df在上面

代码

import findspark
findspark.init('/home/mak/spark-3.0.0-preview2-bin-hadoop2.7')
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('mak').getOrCreate()
import numpy as np
import pandas as pd

# Enable Arrow-based columnar data transfers
spark.conf.set("spark.sql.execution.arrow.enabled", "true")

pdf3 = pd.read_csv('Repayment.csv')
df_repay = spark.createDataFrame(pdf3)

只有加载 df_repay 有问题，其他数据框已成功加载。当我将上面的代码转换为下面的代码时，它成功地工作了

df4 = (spark.read.format("csv").options(header="true") .load("Repayment.csv"))

为什么 df_repay 没有加载 spark.createDataFrame(pdf3) 而相似的数据帧加载成功

Answer 1

pdf3 是 pandas 数据帧，您正在尝试将 pandas 数据帧转换为 spark 数据帧。如果你想坚持你的代码，请使用下面的代码，将你的 pandas 数据帧转换为 spark 数据帧。

from pyspark.sql.types import *
pdf3 = pd.read_csv('Repayment.csv')
#create schema for your dataframe
schema = StructType([StructField("Customer", StringType(), True)\
                   ,StructField("Month", DateType(), True)\
                   ,StructField("Amount", IntegerType(), True)])

#create spark dataframe using schema
df_repay = spark.createDataFrame(pdf3,schema=schema)

TypeError: field Customer: Can not merge type <class 'pyspark.sql.types.StringType'> and <class 'pyspark.sql.types.DoubleType'>

TypeError: field Customer: Can not merge type <class 'pyspark.sql.types.StringType'> and <class 'pyspark.sql.types.DoubleType'>

pandas

apache-spark

pyspark

apache-arrow