1

I am new in SparkR. I am trying to write a query on a dataset using SparkR using Qubole Notebook. several process taken but not getting the output.

Data: Table_A

ID      Name     month      year
1        A         2        2020
2        B         2        2019
3        c        12        2019

Desired Output:

ID      Name     month      year
1        A         2        2020

enter image description here

Variables:

month_value= 2
year_value = 2020

Process 1:

temp_data = sql("
select * 
from Table_A
where month = $month_value and year = $year_value")

Process 2:

temp_data = sql(s"
select * 
from Table_A
where month = $month_value and year = $year_value")

Process 3:

temp_data = sql("
select * 
from Table_A
where month = {0} and year = {1}".format(month_value,year_value))

Process 4:

temp_data = sql("
select * 
from Table_A
where month = ${month_value} and year = ${year_value}")

How to pass the variables in spark SQL, using SparkR?

1 Answer 1

0

The filtering can be done using the where clause in the sql function or using the filter function directly.

Setup with sample data:

library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))
sparkR.session(master = "local[*]", sparkConfig = list(spark.driver.memory = "8g"))

local_df <- data.frame(ID=c(1, 2 ,3),
                       Name=c("A", "B", "c"),
                       month=c(2, 2, 12), 
                       year=c(2020, 2019, 2019))

month_value <- 2
year_value <- 2020

sdf <- createDataFrame(local_df)

First approach using sql. Create the temp view from the SparkDataFrame named sdf. Then you can use glue to compose the sql code. It is a string interpolator which is cleaner than using paste.

# 1) SQL function (string interpolation)
createOrReplaceTempView(sdf, "Table_A")
(sql_expanded <- as.character(glue::glue(
  "select * from Table_A where (month == {month_value}) and (year == {year_value})")))
head(sql(sql_expanded))

The result:

> (sql_expanded <- glue::glue(
+   "select * from Table_A where (month == {month_value}) and (year == {year_value})"))
select * from Table_A where (month == 2) and (year == 2020)
> head(sql(as.character(sql_expanded)))
  ID Name month year
1  1    A     2 2020

The second option, using filter

> head(filter(sdf, sdf$month == month_value & sdf$year == year_value))
  ID Name month year
1  1    A     2 2020

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.