Databricks Error: AnalysisException: Incompatible format detected. with Delta
Databricks Error: AnalysisException: Incompatible format detected. with Delta
当我尝试使用 Databricks 上的 Delta 写入我的数据湖时出现以下错误
fulldf = spark.read.format("csv").option("header", True).option("inferSchema",True).load("/databricks-datasets/flights/")
fulldf.write.format("delta").mode("overwrite").save('/mnt/lake/BASE/flights/Full/')
以上产生以下错误:
AnalysisException: Incompatible format detected.
You are trying to write to `/mnt/lake/BASE/flights/Full/` using Databricks Delta, but there is no
transaction log present. Check the upstream job to make sure that it is writing
using format("delta") and that you are trying to write to the table base path.
To disable this check, SET spark.databricks.delta.formatCheck.enabled=false
To learn more about Delta, see https://docs.databricks.com/delta/index.html
有什么错误原因吗?
当文件夹中有其他格式的数据时,通常会出现此类错误。例如,如果您之前将 Parquet 或 CSV 文件写入其中。完全删除文件夹并重试
这在我类似的情况下有效:
%sql CONVERT TO DELTA parquet.`/mnt/lake/BASE/flights/Full/`
当我尝试使用 Databricks 上的 Delta 写入我的数据湖时出现以下错误
fulldf = spark.read.format("csv").option("header", True).option("inferSchema",True).load("/databricks-datasets/flights/")
fulldf.write.format("delta").mode("overwrite").save('/mnt/lake/BASE/flights/Full/')
以上产生以下错误:
AnalysisException: Incompatible format detected.
You are trying to write to `/mnt/lake/BASE/flights/Full/` using Databricks Delta, but there is no
transaction log present. Check the upstream job to make sure that it is writing
using format("delta") and that you are trying to write to the table base path.
To disable this check, SET spark.databricks.delta.formatCheck.enabled=false
To learn more about Delta, see https://docs.databricks.com/delta/index.html
有什么错误原因吗?
当文件夹中有其他格式的数据时,通常会出现此类错误。例如,如果您之前将 Parquet 或 CSV 文件写入其中。完全删除文件夹并重试
这在我类似的情况下有效:
%sql CONVERT TO DELTA parquet.`/mnt/lake/BASE/flights/Full/`