DataFrames are a fairly new concept to me. A handful of sources recommended it over RDDs and how it outperforms RDDs in many situations. I'd like to see if DataFrames are a viable option for (eventually, I'll be dealing with Array of bytes), so I compared the performance on a word count application on a 1GB file.
Anyways, when I ran the program, I got the following results:
RDD Count total: 137733312 Time Elapsed: 44.5675378 s
DF Count total: 137733312 Time Elapsed: 69.201253448 s
I was expecting the DataFrames to execute faster than the RDD. Is this a result of bad implementation? Or since the DataFrame implementation called textFile, the data was loaded into an RDD and then converted into a DataFrame. Does this impact the performance? Is it recommended to convert my file into say a Parquet file (since that's the default data source) and load directly from it?
I was wondering if someone could explain why RDD's outperformed DataFrames by a pretty significant margin.
def testDF(sc: SparkContext, sqlContext: SQLContext,
fname: String, threshold: Int): Long = {
import sqlContext.implicits._
val linesDF = sc.textFile(fname).toDF("line")
val tokenizer = new Tokenizer().setInputCol("line").setOutputCol("words")
val wordsDF = tokenizer.transform(linesDF)
val countUDF = udf((data: WrappedArray[String]) => data.size)
val countTotal = wordsDF.withColumn("count", countUDF('words)).agg(sum("count"))
countTotal.first()(0).asInstanceOf[Long]
}
def testRDD(sc: SparkContext, fname: String): Int = {
// split each document into words
val tokenized = sc.textFile(fname).flatMap(_.split(" "))
// count the occurrence of each word
val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)
// count characters
val countTotal: Int = wordCounts.map(_._2).reduce((a,b) => a + b)
countTotal
}