Questions tagged [sparkr]
SparkR is an R package that provides a light-weight frontend to use Apache Spark from R.
798
questions
0
votes
0
answers
26
views
Connect to bigquery from databricks using sparklyr
I'm pretty new to databricks. I have it running with Gloogle Cloud.
I have been able to connect to bigquery where I have my tables, and bring some data using SparkR:
library(SparkR)
sparkR.session()
...
0
votes
1
answer
75
views
How to add '.option("overwriteSchema", "true")' to saveAsTable() in SparkR
How can I save an R dataframe with SparkR::saveAsTable() again under the same name as an already existing table after changing columns?
I am working with R on databricks and saved an R dataframe ...
0
votes
0
answers
17
views
How to view full regression summary table from SparkR::spark.logit
If I view a logistic regression model from SparkR::spark.logit() ...
library(SparkR)
mtcars_sdf <- createDataFrame(mtcars)
model <- spark.logit(mtcars_sdf, am ~ wt, family = 'binomial')
summary(...
0
votes
1
answer
55
views
Filter by Multiple Conditions with vectors
I am trying to filter by multiple variables for a series of conditions using vectors on a SparkR dataframe.
Doing this with tidyverse on a regular dataframe is straight forward. For example:
library(&...
0
votes
0
answers
60
views
how to install spark and configure spark standalone cluster in ubuntu 22.04 and connect it from RStudio installed on windows machine
I have installed spark on Ubuntu (remote server) but I am unable to connect it from rstudio that is installed on my local machine(windows os).
Error in start_shell(master = master, spark_home = ...
0
votes
1
answer
66
views
Add leading zeros to character column in a Spark data frame using sparklyr
How can I add leading zeroes to data$var1 in the following MWE?
library(sparklyr)
data <- data.frame(
var1 = c("ab", "abc", "abcd"),
var2 = c(1, 2, 3)
)
data <- ...
0
votes
0
answers
9
views
SparkR functions failing in databricks
I'm not an R user, but i've had a need to use SparkR for some analysis and i am trying to run the code below. Are pipes the issue here?
table %>%
SparkR::select('station') %>%
SparkR::...
1
vote
1
answer
233
views
"Error in validVersions[with(validVersions, order(hadoop, decreasing = TRUE)), : subscript out of bounds" when using spark_install
When I use spark_isntall("3.0"), I get an error (screenshot)
Error in validVersions[with(validVersions, order(hadoop, decreasing = TRUE)), : subscript out of bounds
I re-installed R and ...
1
vote
1
answer
257
views
Databricks: configure R to use custom CRAN mirror by default
I want to specify a default custom CRAN mirror in R under Databricks, but adjusting the config in the Rprofile.site file seems not to bet recognized at all.
I have already read the official Microsoft ...
0
votes
0
answers
65
views
How to read a table from snowflake using R scripting in Databricks
I am trying to read data from snowflake to Databricks using R scripting. Can anyone please help how to write. I could not find it in documentation.
0
votes
1
answer
80
views
Sparklyr / Dplyr Solution for counting occurrences of each value in all columns to one table
I would like to get a table from my data where the first column has a list of values that show up in the data and each additional column corresponds to a column in the data and gives the number of ...
0
votes
0
answers
26
views
r code giving error coerce type 'S4' to vector of type double
I am new to r code so getting some issue here is the code
library(sparkr)
library(arulesViz)
test1 <- "select t.result from usertest l join userlab t on t.uid=l.uid where t.result is not null&...
1
vote
0
answers
58
views
Unable to enable SparkR arrow
This is the code that I used to enable Arrow
library(SparkR)
Sys.setenv(SPARK_HOME = "/home/spark/spark-3.3.2-bin-hadoop3")
install.packages("arrow", repos="https://cloud.r-...
0
votes
0
answers
88
views
"cannot open compressed file" with Databricks SparkR trying to read RDS from S3
I am trying to use readRDS in Databricks notebook, reading a file in S3.
I have the complete S3 path and .rds object name in "S3_path".
rds <- readRDS(S3_path)
I get the error:
Warning ...
0
votes
0
answers
35
views
Create table on SQL server with SparkR
From R notebook on Databricks I would like to create a table on SQL server. I am able to read SQL tables with the following:
library('magrittr')
library('SparkR')
conn = "jdbc:sqlserver://...
0
votes
0
answers
48
views
Spark not found in the cache directory. Installation will start. MirrorUrl not provided
I am trying to use SparkR in R...however it gives an error for the following line of code
install.spark(hadoopVersion = "2.7", mirrorUrl = NULL, localDir = NULL, overwrite = FALSE)
This is ...
0
votes
2
answers
81
views
Error in getSparkSession() : SparkSession not initialized
I ran a single line of code to make a Spark DataFrame. I installed SparkR and used library(SparkR) before I ran the following:
spark_df <- as.DataFrame(data)
However I get the following error ...
0
votes
1
answer
49
views
Error in sparkCheckInstall(sparkHome, master, deployMode):
After doing the following
devtools::install_github('apache/[email protected]', subdir='R/pkg', force = TRUE)
library(SparkR)
I ran this to convert my data into a spark DataFrame
as.DataFrame(value1)
...
0
votes
0
answers
52
views
trying to convert R code to sparkR code facing error
hi i want to convert the following code written in R, to run as a SparkR code-
data_yoy <- selected_columns2 %>%
dplyr::group_by_at(GROUP2) %>%
dplyr::arrange(metric) %>%
dplyr::mutate(...
0
votes
1
answer
101
views
group by and concatenate string using sparklyr
There are a number of questions asking precisely the same thing but none within the context of a sparklyr environment. How does one group by a column and then concatenate the values of some other ...
0
votes
0
answers
44
views
Is there a way to use R Fable with Spark?
I've successfully built a pipeline for hierarchial time-series forecasting using the fable package. For now I've used multicore function from the future pacakge (in Databricks). And, for ~1k time ...
0
votes
0
answers
31
views
SparkR logging method
What is the logging method for SparkR, the equivalent of SparklyR spark_log(sc, -1) method
E.g:get the SparklyR and in SparkR context
sc <- get_spark_connection()
sparkR.session()
#set the logging ...
0
votes
0
answers
105
views
Databricks SparkR can't create spark DataFrame from large R data.frame
I have a local R data.frame with 16 Million records in databricks. When I try to load it into a spark DataFrame I receive an error after circa 5 minutes every time.
sparkR.session()
spark_data_frame &...
1
vote
1
answer
174
views
Convert a list of fields to structtype object which is a SparkR schema
We have to get the schema of dataframe in SparkR as StructType and list as list of fields, e.g:
str(schema)
#List of 2
# $ jobj :Class 'jobj' <environment: 0x563114ff5900>
# $ fields:function (...
1
vote
1
answer
111
views
Change type from AsIs to list in R dataframe
I'm trying to make a dataframe in R that looks like this:
group numbers
1 1 [1,2,3,4,5]
2 2 [8,9,10,11]
The way I went about it was as follows:
group = c(1:2)
numbers = I(list((1:...
2
votes
1
answer
205
views
Use multiple spark connections in a databricks notebook
I want to connect to two database tables in a single databricks (R) notebook using spark and sparklyr:
sc_one <- spark_connect(method="databricks")
tbl_change_db(sc_one, "...
0
votes
1
answer
262
views
Using sparklyr write SQL on new Spark Dataframe
I am able to create a Spark dataframe using:
df_01 <- sdf_sql(sc, "select * from database.table")
How can I use sparklyr to access the created Spark dataframe df_01 in a subsequent SQL ...
0
votes
1
answer
62
views
Sparklyr/Spark NLP connect via YARN
I'm new to sparklyr and spark nlp. Had gotten a local connection running no problem and test data was saving and being read back etc. Today when I loaded the real data which is a batch of text data ...
0
votes
1
answer
79
views
Options for inserting large dataset from databricks to SQL table using sparkR
I am trying to write a large dataset (millions) to an SQL table (Impala) using sparkR in databricks. I have found two options, neither of which are working.
Writing using a simple insertInto fails ...
0
votes
1
answer
252
views
Is there a more memory efficient way to concatenate two columns in SparkR?
I'm trying to concatenate two columns from two spark data frames using SparkR e.g.
Frame1
ID
Data1
Data2
1
0
0
2
1
0
3
1
1
4
0
1
5
1
1
Frame2
OtherID
Data1
Data2
2
0
0
3
1
0
4
1
1
5
0
1
6
...
0
votes
1
answer
43
views
Filter Sparkdataframe using values from another dataframe R
I have a sparkdataframe that I need to filter based on an ID column which is present in a different dataframe. I am trying to pass this as a list and it doesn't work. Can anyone help ?
head(filter(DF, ...
1
vote
1
answer
560
views
Convert a sql result table to a spark R dataframe - Databricks
I am running a notebook on databricks for an analysis. As part of that, I have a SQL query that produces a df which I want to convert to R df for further analysis. I tried saving that as a temp, but ...
0
votes
1
answer
55
views
SparkR: custom aggregete function
I need to calculate aggregate using a native R function IQR.
df1 <- SparkR::createDataFrame(iris)
df2 <- SparkR::agg(SparkR::groupBy(df1, "Species"),
IQR_Sepal_Length=IQR(df1$...
0
votes
1
answer
82
views
Running sparklyr Functions in Parallel
I am currently using sparklyr to transform a large dataset and calculate a set of metrics based on an input start and end date. This function is run on the same dataframe for multiple start and end ...
1
vote
1
answer
72
views
Error when using SparkR insertInto via databricks
I am trying to insert values from a dataframe into a database table (impala) using SparkR in a databricks notebook:
require(SparkR)
test_df <- data.frame(row_no = c(2,3,4,5,6,7,8)
...
0
votes
1
answer
43
views
SparkR: compare schemas of sparkdataframes
For example, we have 2 two identical sparkdataframes
library(SparkR)
df1 <- createDataFrame(iris)
df2 <- createDataFrame(iris)
How can we check that they have identical schemas?
sdf1 <- ...
0
votes
1
answer
100
views
SparkR and dplyr: window function count() using gapply
I'm trying to implement simple query on Spark using "gapply", but face troubles.
This code works well.
library(SparkR)
library(dplyr)
df <- createDataFrame(iris)
createOrReplaceTempView(...
0
votes
1
answer
426
views
Arrow R package fails to install on Databricks
About 6 weeks ago (early April 2022), I had tested a Databricks workflow to ensure that I could trigger jobs on databricks remotely from Airflow, which was successful.
As part of the process the ...
0
votes
2
answers
1k
views
How to Read Append Blobs as DataFrames in Azure DataBricks
My batch processing pipeline in Azure has the following scenario: I am using the copy activity in Azure Data Factory to unzip thousands of zip files, stored in a blob storage container. These zip ...
0
votes
2
answers
116
views
How to save an empty data to adlsgen2 using R?
I've explored but didn't find any suggestions.
I want to know how to save an empty file to adlsgen2 using R and then read it back to the same code.
Thanks for the help.
1
vote
1
answer
816
views
Databricks: how to convert Spark dataframe under %r to dataframe under %python
I found some tips about converting a pyspark dataframe to R, but I need to perform the opposite task: convert a R dataframe to pyspark
Anyone knows how to do it?
0
votes
1
answer
154
views
Running Prophet with SparkDataFrame in R in Databricks
I have a Spark Dataframe in R. I retrieved the dataset from snowflake by doing the following:
snowflake_tbl_name <- "xxxxxx"
tickerDF <- SparkR::read.df(
...
1
vote
1
answer
162
views
What is the issue with my pagerank algorithm code?
df1 \<- tbl(sc ,"databricks.tablename")
g \<- graph_from_data_frame(df1, directed = TRUE, vertices = NULL) # Directed Igraph creation
is_weighted(g) #Check for Weight
pagerank \<-...
0
votes
0
answers
122
views
Java Heap Space Error in SparklyR & Tidymodels
Hi,
I am trying to model a large dataset (2000000+ rows, ~150 columns) using the tidymodels and the spark engine. However, I am receiving the following error: java.lang.OutOfMemoryError: Java heap ...
0
votes
1
answer
66
views
How to drop all NA columns in a SparkDataFrame with SparkR?
Once again, I'm facing a problem that I can't transcribe under SparkR.
I have a SparkDataFrame which some columns contain only NAs, and I want to delete all these columns.
I discovered SparkR recently,...
0
votes
2
answers
100
views
How to add multiple calculated columns to a SparkDataFrame using SparkR?
Now I'm stuck on a rather basic case, but I can't find a clever solution with SparkR ...
From N columns in my SparkDataFrame, I need to create N new calculated columns.
df <- data.frame(V1 = base::...
1
vote
1
answer
73
views
sparklyr: fill `spark_read_parquet path` argument from list
I'd like to process several files in the same s3 bucket in the same way. So, I created a list of the filenames:
dt <- seq(as.Date("1991/12/23"), by = "day", length.out = 5)
...
0
votes
1
answer
149
views
How to use an undefined list of variables as column names in a SparkDataFrame with SparkR?
I keep progressing in the world of SparkR, and I am now facing an issue which I am unable to resolve.
Working on SparkDataFrame manipulations, I may want to update some columns, or aggregate others. I ...
0
votes
1
answer
239
views
Apply a function to multiple columns of a SparkDataFrame, at once
In basic R, this is how I apply a function to multiple columns at once, using data.table :
d <- data.table(V1 = rep(1:2, 4:3), V2 = c(1, 2, 4, 5, 2, 3, 4), V3 = 1:7, V4 = sample(letters, 7))
Cols &...
2
votes
2
answers
178
views
How to store SparkR result into an R object?
Still new to the world of Azure Databricks, the use of SparkR remains very obscure to me, even for very simple tasks...
It took me a very long time to find how to count distinct values, and I'm not ...