没有规定将 Spark DataFrame 转换为 scala 中的 AWS Glue DynamicFrame
No provision to convert Spark DataFrame to AWS Glue DynamicFrame in scala
不存在从 Spark DataFrame 转换为 Glue DynamicFrame 的等效代码,是否是故意的,解决方法是什么?
# Convert to a dataframe and partition based on "partition_col"
partitioned_dataframe = datasource0.toDF().repartition(1)
# Convert back to a DynamicFrame for further processing.
partitioned_dynamicframe = DynamicFrame.fromDF(partitioned_dataframe, glueContext, "partitioned_df")
使用这个:
# Convert Spark's DataFrame to Glue's DynamicFrame
val partitioned_dynamicframe = DynamicFrame(partitioned_dataframe, glueContext)
不存在从 Spark DataFrame 转换为 Glue DynamicFrame 的等效代码,是否是故意的,解决方法是什么?
# Convert to a dataframe and partition based on "partition_col"
partitioned_dataframe = datasource0.toDF().repartition(1)
# Convert back to a DynamicFrame for further processing.
partitioned_dynamicframe = DynamicFrame.fromDF(partitioned_dataframe, glueContext, "partitioned_df")
使用这个:
# Convert Spark's DataFrame to Glue's DynamicFrame
val partitioned_dynamicframe = DynamicFrame(partitioned_dataframe, glueContext)