15

I have a Spark DataFrame df with five columns. I want to add another column with its values being the tuple of the first and second columns. When using with withColumn() method, I get the mismatch error, because the input is not Column type, but instead (Column,Column). I wonder if there is a solution beside running for loop over the rows in this case?

var dfCol=(col1:Column,col2:Column)=>(col1,col2)
val vv = df.withColumn( "NewColumn", dfCol( df(df.schema.fieldNames(1)) , df(df.schema.fieldNames(2)) ) )

4 Answers 4

30

You can use struct function which creates a tuple of provided columns:

import org.apache.spark.sql.functions.struct

val df = Seq((1,2), (3,4), (5,3)).toDF("a", "b")
df.withColumn("NewColumn", struct(df("a"), df("b")).show(false)

+---+---+---------+
|a  |b  |NewColumn|
+---+---+---------+
|1  |2  |[1,2]    |
|3  |4  |[3,4]    |
|5  |3  |[5,3]    |
+---+---+---------+
3
  • What if I want to delete the existing a and b columns? How to do that please?
    – Soumendra
    Apr 9, 2019 at 12:58
  • 1
    @Soumendra df.drop('a'). Apr 26, 2019 at 13:35
  • This is the better way of doing it. Verified.
    – Innovation
    Feb 12, 2021 at 7:56
13

You can use a User-defined function udf to achieve what you want.

UDF definition

object TupleUDFs {
  import org.apache.spark.sql.functions.udf      
  // type tag is required, as we have a generic udf
  import scala.reflect.runtime.universe.{TypeTag, typeTag}

  def toTuple2[S: TypeTag, T: TypeTag] = 
    udf[(S, T), S, T]((x: S, y: T) => (x, y))
}

Usage

df.withColumn(
  "tuple_col", TupleUDFs.toTuple2[Int, Int].apply(df("a"), df("b"))
)

assuming "a" and "b" are the columns of type Int you want to put in a tuple.

1
  • @TNM: Your edit got rejected unfortunately, as edit comments did not state clearly about wrong import, I used. It is corrected now. Sep 27, 2015 at 9:05
6

You can merge multiple dataframe columns into one using array.

// $"*" will capture all existing columns
df.select($"*", array($"col1", $"col2").as("newCol")) 
2

If you want to merge two dataframe columns into one column. Just:

import org.apache.spark.sql.functions.array
df.withColumn("NewColumn", array("columnA", "columnB"))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.