0

I have a code snippit like this:

val filteredDF = df.filter($"abc.color".equalTo("yellow") && $"abc.height".equalTo("5")).toDF()

The complex struct looks like:

{
  abc:{
      color: yellow,
      height:5
  }
}

What I want is to create a variable to represent abc, say

val table_name = "abc"

and create variables for color and height as well, say

val colorField = "color"
val heightField = "height"

Then how can I modify the code above to use those new variables?

I tried some ways like

df.filter(${table_name+"."+colorField}.equalTo("yellow")

and

df.filter(($""+table_name+"."+colorField).equalTo("yellow")

But neither works.

1 Answer 1

1

You can use Scala's String Interpolation

val table_name = "abc"
val colorField = "color"
val heightField = "height"

df.show
+----------+
|       abc|
+----------+
|[yellow,5]|
+----------+

import org.apache.spark.sql.functions.col

df.filter(col(s"${table_name}.${colorField}").equalTo("yellow")).show

+----------+
|       abc|
+----------+
|[yellow,5]|
+----------+

df.filter(col(s"${table_name}.${colorField}").equalTo("green")).show
+---+
|abc|
+---+
+---+
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.