29

I have a dataframe in Spark with many columns and a udf that I defined. I want the same dataframe back, except with one column transformed. Furthermore, my udf takes in a string and returns a timestamp. Is there an easy way to do this? I tried

val test = myDF.select("my_column").rdd.map(r => getTimestamp(r)) 

but this returns an RDD and just with the transformed column.

0

1 Answer 1

48

If you really need to use your function, I can suggest two options:

  1. Using map / toDF:

    import org.apache.spark.sql.Row
    import sqlContext.implicits._
    
    def getTimestamp: (String => java.sql.Timestamp) = // your function here
    
    val test = myDF.select("my_column").rdd.map {
      case Row(string_val: String) => (string_val, getTimestamp(string_val))
    }.toDF("my_column", "new_column")
    
  2. Using UDFs (UserDefinedFunction):

    import org.apache.spark.sql.functions._
    
    def getTimestamp: (String => java.sql.Timestamp) = // your function here
    
    val newCol = udf(getTimestamp).apply(col("my_column")) // creates the new column
    val test = myDF.withColumn("new_column", newCol) // adds the new column to original DF
    

Alternatively,

If you just want to transform a StringType column into a TimestampType column you can use the unix_timestamp column function available since Spark SQL 1.5:

val test = myDF
  .withColumn("new_column", unix_timestamp(col("my_column"), "yyyy-MM-dd HH:mm")
  .cast("timestamp"))

Note: For spark 1.5.x, it is necessary to multiply the result of unix_timestamp by 1000 before casting to timestamp (issue SPARK-11724). The resulting code would be:

val test = myDF
  .withColumn("new_column", (unix_timestamp(col("my_column"), "yyyy-MM-dd HH:mm") *1000L)
  .cast("timestamp"))

Edit: Added udf option

2
  • 1
    Thanks for the help. The only issue i'm having is that when I do df.withColumn("start_date", unix_timestamp(df1("start_date"), "yyyy-MM-dd HH:mm:ss").cast("timestamp")), my dates are getting converted wrong. For example: 2013-08-12 06:40:54 gets converted to 1970-01-16 22:18:09.654. Do you happen to know what might be going on?
    – mt88
    Commented May 5, 2016 at 18:24
  • 1
    For spark 1.5 you must multiply by 1000 before the cast Commented May 5, 2016 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.