SparkR Stage X 包含一个非常大的任务

Question

我在使用 R 数据框调用 createOrReplaceTempView 时收到此警告：

createOrReplaceTempView (as.Data.Frame(products), "prod")

我应该忽略这个警告吗？这效率低下？

谢谢！

Answer 1

这些只是警告。如果您想尝试避免它们，请重新分区数据并在注册临时 table 并对数据执行某些功能之前对其调用操作。重新分区会导致洗牌。

例如，

set.seed(123)
df<- data.frame(thing1=rnorm(100000), thing2=rep("ThisIsAString", 100000), stringsAsFactors = FALSE)
sdf<- SparkR::createDataFrame(df) # Warnings for me
SparkR::getNumPartitions(sdf) # 1 partition
sdf<- SparkR::repartition(sdf, numPartitions=4L) # repartition, will cause a shuffle
SparkR::getNumPartitions(sdf) # spark now knows to repartition the data, this will happen once an action is called on the data, i.e. counting the rows
SparkR::cache(sdf) # Nothing has happened yet
SparkR::nrow(sdf) # Now cause the repartition and a count to happen. # Will be warned
SparkR::createOrReplaceTempView(sdf, "sdfTable") # Make a temp table as you have in your example

res<- SparkR::sql("SELECT thing1, thing2 FROM sdfTable WHERE thing1> 0.5") # SQL
SparkR::nrow(res) # no warnings, 31002 observations found. 
SparkR::getNumPartitions(res) # 4 partitions in the result

SparkR Stage X 包含一个非常大的任务

SparkR Stage X contains a task of very large size

apache-spark

sparkr