8

Is there any way that I can evaluate my Column expression if I am only using Literal (no dataframe columns).

For example, something like:

val result: Int = someFunction(lit(3) * lit(5))
//result: Int = 15

or

import org.apache.spark.sql.function.sha1
val result: String = someFunction(sha1(lit("5")))
//result: String = ac3478d69a3c81fa62e60f5c3696165a4e5e6ac4

I am able to evaluate using a dataframes

val result = Seq(1).toDF.select(sha1(lit("5"))).as[String].first
//result: String = ac3478d69a3c81fa62e60f5c3696165a4e5e6ac4

But is there any way to get the same results without using dataframe?

2
  • 1
    Just curious, why do you need this? Also I tried your approach and it does not trigger any spark job. Therefore could you tell us what's wrong with the approach you propose?
    – Oli
    Jul 6, 2018 at 12:31
  • There are so many use cases like to test your custom UDF's, to test your complex column expressions and to reuse existing functions from sql.function._ in your custom UDF's.
    – Kaushal
    Jul 10, 2018 at 9:16

2 Answers 2

10

To evaluate a literal column you can convert it to an Expression and eval without providing input row:

scala> sha1(lit("1").cast("binary")).expr.eval()
res1: Any = 356a192b7913b04c54574d18c28d46e6395428ab

As long as the function is an UserDefinedFunction it will work the same way:

scala> val f = udf((x: Int) => x)
f: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,Some(List(IntegerType)))

scala> f(lit(3) * lit(5)).expr.eval()
res3: Any = 15
1
  • Very cool stuff! Do you know how to make it work with string expressions? expr("sha1(CAST(1 AS BINARY))").expr.eval() fails with "java.lang.UnsupportedOperationException: Cannot evaluate expression: 'sha1(cast(1 as binary))"
    – Zohar Meir
    Sep 22, 2022 at 12:11
-2

The following code can help:

val isUuid = udf((uuid: String) => uuid.matches("[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}"))


df.withColumn("myCol_is_uuid",isUuid(col("myCol")))
.filter("myCol_is_uuid = true")
.show(10, false)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.