1

I would like to convert a RDD containing records of strings, like below, to a Spark dataframe.

"Mike,2222-003330,NY,34"
"Kate,3333-544444,LA,32"
"Abby,4444-234324,MA,56"
....

The schema line is not inside the same RDD, but in a another variable:

val header = "name,account,state,age"

So now my question is, how do I use the above two, to create a dataframe in Spark? I am using Spark version 2.2.

I did search and saw a post: Can I read a CSV represented as a string into Apache Spark using spark-csv . However it's not exactly what I need and I can't figure out a way to modify this piece of code to work in my case.

Your help is greatly appreciated.

1 Answer 1

2

The easier way would probably be to start from the CSV file and read it directly as a dataframe (by specifying the schema). You can see an example here: Provide schema while reading csv file as a dataframe.


When the data already exists in an RDD you can use toDF() to convert to a dataframe. This function also accepts column names as input. To use this functionality, first import the spark implicits using the SparkSession object:

val spark: SparkSession = SparkSession.builder.getOrCreate()
import spark.implicits._

Since the RDD contains strings it needs to first be converted to tuples representing the columns in the dataframe. In this case, this will be a RDD[(String, String, String, Int)] since there are four columns (the last age column is changed to int to illustrate how it can be done).

Assuming the input data are in rdd:

val header = "name,account,state,age"

val df = rdd.map(row => row.split(","))
  .map{ case Array(name, account, state, age) => (name, account, state, age.toInt)}
  .toDF(header.split(","):_*)

Resulting dataframe:

+----+-----------+-----+---+
|name|    account|state|age|
+----+-----------+-----+---+
|Mike|2222-003330|   NY| 34|
|Kate|3333-544444|   LA| 32|
|Abby|4444-234324|   MA| 56|
+----+-----------+-----+---+
2
  • 1
    Thank you very much this is exactly what I need !! May 3, 2018 at 0:43
  • Maybe this is ok when there's just a couple of lines that you can visually inspect, but generally it's not idea to parse CSV using simple split by newline and comma, because you can have those characters inside a cell value. CSV is "parsed", which is fundamentally different from simple text manipulation and even from regex pattern matching, it's preferred to use a real CSV library.
    – Ion Lesan
    May 14, 2021 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.