Questions tagged [sparkr]

SparkR is an R package that provides a light-weight frontend to use Apache Spark from R.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Connect to bigquery from databricks using sparklyr

I'm pretty new to databricks. I have it running with Gloogle Cloud. I have been able to connect to bigquery where I have my tables, and bring some data using SparkR: library(SparkR) sparkR.session() ...
filemonPi's user avatar
0 votes
1 answer
75 views

How to add '.option("overwriteSchema", "true")' to saveAsTable() in SparkR

How can I save an R dataframe with SparkR::saveAsTable() again under the same name as an already existing table after changing columns? I am working with R on databricks and saved an R dataframe ...
karara's user avatar
  • 1
0 votes
0 answers
17 views

How to view full regression summary table from SparkR::spark.logit

If I view a logistic regression model from SparkR::spark.logit() ... library(SparkR) mtcars_sdf <- createDataFrame(mtcars) model <- spark.logit(mtcars_sdf, am ~ wt, family = 'binomial') summary(...
Joe's user avatar
  • 3,313
0 votes
1 answer
55 views

Filter by Multiple Conditions with vectors

I am trying to filter by multiple variables for a series of conditions using vectors on a SparkR dataframe. Doing this with tidyverse on a regular dataframe is straight forward. For example: library(&...
daszlosek's user avatar
  • 1,406
0 votes
0 answers
60 views

how to install spark and configure spark standalone cluster in ubuntu 22.04 and connect it from RStudio installed on windows machine

I have installed spark on Ubuntu (remote server) but I am unable to connect it from rstudio that is installed on my local machine(windows os). Error in start_shell(master = master, spark_home = ...
Hikmat Saud's user avatar
0 votes
1 answer
66 views

Add leading zeros to character column in a Spark data frame using sparklyr

How can I add leading zeroes to data$var1 in the following MWE? library(sparklyr) data <- data.frame( var1 = c("ab", "abc", "abcd"), var2 = c(1, 2, 3) ) data <- ...
Christoffer's user avatar
0 votes
0 answers
9 views

SparkR functions failing in databricks

I'm not an R user, but i've had a need to use SparkR for some analysis and i am trying to run the code below. Are pipes the issue here? table %>% SparkR::select('station') %>% SparkR::...
ZachW's user avatar
  • 48
1 vote
1 answer
233 views

"Error in validVersions[with(validVersions, order(hadoop, decreasing = TRUE)), : subscript out of bounds" when using spark_install

When I use spark_isntall("3.0"), I get an error (screenshot) Error in validVersions[with(validVersions, order(hadoop, decreasing = TRUE)), : subscript out of bounds I re-installed R and ...
Justin Liang's user avatar
1 vote
1 answer
257 views

Databricks: configure R to use custom CRAN mirror by default

I want to specify a default custom CRAN mirror in R under Databricks, but adjusting the config in the Rprofile.site file seems not to bet recognized at all. I have already read the official Microsoft ...
Niklas Letz's user avatar
0 votes
0 answers
65 views

How to read a table from snowflake using R scripting in Databricks

I am trying to read data from snowflake to Databricks using R scripting. Can anyone please help how to write. I could not find it in documentation.
Adi's user avatar
  • 21
0 votes
1 answer
80 views

Sparklyr / Dplyr Solution for counting occurrences of each value in all columns to one table

I would like to get a table from my data where the first column has a list of values that show up in the data and each additional column corresponds to a column in the data and gives the number of ...
34jbonz's user avatar
  • 118
0 votes
0 answers
26 views

r code giving error coerce type 'S4' to vector of type double

I am new to r code so getting some issue here is the code library(sparkr) library(arulesViz) test1 <- "select t.result from usertest l join userlab t on t.uid=l.uid where t.result is not null&...
patdev's user avatar
  • 11
1 vote
0 answers
58 views

Unable to enable SparkR arrow

This is the code that I used to enable Arrow library(SparkR) Sys.setenv(SPARK_HOME = "/home/spark/spark-3.3.2-bin-hadoop3") install.packages("arrow", repos="https://cloud.r-...
Vortenzie's user avatar
0 votes
0 answers
88 views

"cannot open compressed file" with Databricks SparkR trying to read RDS from S3

I am trying to use readRDS in Databricks notebook, reading a file in S3. I have the complete S3 path and .rds object name in "S3_path". rds <- readRDS(S3_path) I get the error: Warning ...
Chuck's user avatar
  • 1,137
0 votes
0 answers
35 views

Create table on SQL server with SparkR

From R notebook on Databricks I would like to create a table on SQL server. I am able to read SQL tables with the following: library('magrittr') library('SparkR') conn = "jdbc:sqlserver://...
user1700890's user avatar
  • 7,276
0 votes
0 answers
48 views

Spark not found in the cache directory. Installation will start. MirrorUrl not provided

I am trying to use SparkR in R...however it gives an error for the following line of code install.spark(hadoopVersion = "2.7", mirrorUrl = NULL, localDir = NULL, overwrite = FALSE) This is ...
Vortenzie's user avatar
0 votes
2 answers
81 views

Error in getSparkSession() : SparkSession not initialized

I ran a single line of code to make a Spark DataFrame. I installed SparkR and used library(SparkR) before I ran the following: spark_df <- as.DataFrame(data) However I get the following error ...
Vortenzie's user avatar
0 votes
1 answer
49 views

Error in sparkCheckInstall(sparkHome, master, deployMode):

After doing the following devtools::install_github('apache/[email protected]', subdir='R/pkg', force = TRUE) library(SparkR) I ran this to convert my data into a spark DataFrame as.DataFrame(value1) ...
Vortenzie's user avatar
0 votes
0 answers
52 views

trying to convert R code to sparkR code facing error

hi i want to convert the following code written in R, to run as a SparkR code- data_yoy <- selected_columns2 %>% dplyr::group_by_at(GROUP2) %>% dplyr::arrange(metric) %>% dplyr::mutate(...
ritzen101's user avatar
0 votes
1 answer
101 views

group by and concatenate string using sparklyr

There are a number of questions asking precisely the same thing but none within the context of a sparklyr environment. How does one group by a column and then concatenate the values of some other ...
Cyrus Mohammadian's user avatar
0 votes
0 answers
44 views

Is there a way to use R Fable with Spark?

I've successfully built a pipeline for hierarchial time-series forecasting using the fable package. For now I've used multicore function from the future pacakge (in Databricks). And, for ~1k time ...
Brian Head's user avatar
0 votes
0 answers
31 views

SparkR logging method

What is the logging method for SparkR, the equivalent of SparklyR spark_log(sc, -1) method E.g:get the SparklyR and in SparkR context sc <- get_spark_connection() sparkR.session() #set the logging ...
carmel's user avatar
  • 13
0 votes
0 answers
105 views

Databricks SparkR can't create spark DataFrame from large R data.frame

I have a local R data.frame with 16 Million records in databricks. When I try to load it into a spark DataFrame I receive an error after circa 5 minutes every time. sparkR.session() spark_data_frame &...
Sanchez333's user avatar
1 vote
1 answer
174 views

Convert a list of fields to structtype object which is a SparkR schema

We have to get the schema of dataframe in SparkR as StructType and list as list of fields, e.g: str(schema) #List of 2 # $ jobj :Class 'jobj' <environment: 0x563114ff5900> # $ fields:function (...
carmel's user avatar
  • 13
1 vote
1 answer
111 views

Change type from AsIs to list in R dataframe

I'm trying to make a dataframe in R that looks like this: group numbers 1 1 [1,2,3,4,5] 2 2 [8,9,10,11] The way I went about it was as follows: group = c(1:2) numbers = I(list((1:...
iYOA's user avatar
  • 130
2 votes
1 answer
205 views

Use multiple spark connections in a databricks notebook

I want to connect to two database tables in a single databricks (R) notebook using spark and sparklyr: sc_one <- spark_connect(method="databricks") tbl_change_db(sc_one, "...
Sanchez333's user avatar
0 votes
1 answer
262 views

Using sparklyr write SQL on new Spark Dataframe

I am able to create a Spark dataframe using: df_01 <- sdf_sql(sc, "select * from database.table") How can I use sparklyr to access the created Spark dataframe df_01 in a subsequent SQL ...
Scott's user avatar
  • 446
0 votes
1 answer
62 views

Sparklyr/Spark NLP connect via YARN

I'm new to sparklyr and spark nlp. Had gotten a local connection running no problem and test data was saving and being read back etc. Today when I loaded the real data which is a batch of text data ...
Magnetar's user avatar
  • 105
0 votes
1 answer
79 views

Options for inserting large dataset from databricks to SQL table using sparkR

I am trying to write a large dataset (millions) to an SQL table (Impala) using sparkR in databricks. I have found two options, neither of which are working. Writing using a simple insertInto fails ...
Sanchez333's user avatar
0 votes
1 answer
252 views

Is there a more memory efficient way to concatenate two columns in SparkR?

I'm trying to concatenate two columns from two spark data frames using SparkR e.g. Frame1 ID Data1 Data2 1 0 0 2 1 0 3 1 1 4 0 1 5 1 1 Frame2 OtherID Data1 Data2 2 0 0 3 1 0 4 1 1 5 0 1 6 ...
Yoddlenod's user avatar
0 votes
1 answer
43 views

Filter Sparkdataframe using values from another dataframe R

I have a sparkdataframe that I need to filter based on an ID column which is present in a different dataframe. I am trying to pass this as a list and it doesn't work. Can anyone help ? head(filter(DF, ...
Datamaniac's user avatar
1 vote
1 answer
560 views

Convert a sql result table to a spark R dataframe - Databricks

I am running a notebook on databricks for an analysis. As part of that, I have a SQL query that produces a df which I want to convert to R df for further analysis. I tried saving that as a temp, but ...
Datamaniac's user avatar
0 votes
1 answer
55 views

SparkR: custom aggregete function

I need to calculate aggregate using a native R function IQR. df1 <- SparkR::createDataFrame(iris) df2 <- SparkR::agg(SparkR::groupBy(df1, "Species"), IQR_Sepal_Length=IQR(df1$...
Jovanny's user avatar
  • 23
0 votes
1 answer
82 views

Running sparklyr Functions in Parallel

I am currently using sparklyr to transform a large dataset and calculate a set of metrics based on an input start and end date. This function is run on the same dataframe for multiple start and end ...
Dave Wilson's user avatar
1 vote
1 answer
72 views

Error when using SparkR insertInto via databricks

I am trying to insert values from a dataframe into a database table (impala) using SparkR in a databricks notebook: require(SparkR) test_df <- data.frame(row_no = c(2,3,4,5,6,7,8) ...
Sanchez333's user avatar
0 votes
1 answer
43 views

SparkR: compare schemas of sparkdataframes

For example, we have 2 two identical sparkdataframes library(SparkR) df1 <- createDataFrame(iris) df2 <- createDataFrame(iris) How can we check that they have identical schemas? sdf1 <- ...
Jovanny's user avatar
  • 23
0 votes
1 answer
100 views

SparkR and dplyr: window function count() using gapply

I'm trying to implement simple query on Spark using "gapply", but face troubles. This code works well. library(SparkR) library(dplyr) df <- createDataFrame(iris) createOrReplaceTempView(...
Jovanny's user avatar
  • 23
0 votes
1 answer
426 views

Arrow R package fails to install on Databricks

About 6 weeks ago (early April 2022), I had tested a Databricks workflow to ensure that I could trigger jobs on databricks remotely from Airflow, which was successful. As part of the process the ...
APW's user avatar
  • 31
0 votes
2 answers
1k views

How to Read Append Blobs as DataFrames in Azure DataBricks

My batch processing pipeline in Azure has the following scenario: I am using the copy activity in Azure Data Factory to unzip thousands of zip files, stored in a blob storage container. These zip ...
khidir sanosi's user avatar
0 votes
2 answers
116 views

How to save an empty data to adlsgen2 using R?

I've explored but didn't find any suggestions. I want to know how to save an empty file to adlsgen2 using R and then read it back to the same code. Thanks for the help.
Manish Kumar Biswas's user avatar
1 vote
1 answer
816 views

Databricks: how to convert Spark dataframe under %r to dataframe under %python

I found some tips about converting a pyspark dataframe to R, but I need to perform the opposite task: convert a R dataframe to pyspark Anyone knows how to do it?
Thais Guerra Braga's user avatar
0 votes
1 answer
154 views

Running Prophet with SparkDataFrame in R in Databricks

I have a Spark Dataframe in R. I retrieved the dataset from snowflake by doing the following: snowflake_tbl_name <- "xxxxxx" tickerDF <- SparkR::read.df( ...
nak5120's user avatar
  • 4,119
1 vote
1 answer
162 views

What is the issue with my pagerank algorithm code?

df1 \<- tbl(sc ,"databricks.tablename") g \<- graph_from_data_frame(df1, directed = TRUE, vertices = NULL) # Directed Igraph creation is_weighted(g) #Check for Weight pagerank \<-...
Anand Salimath's user avatar
0 votes
0 answers
122 views

Java Heap Space Error in SparklyR & Tidymodels

Hi, I am trying to model a large dataset (2000000+ rows, ~150 columns) using the tidymodels and the spark engine. However, I am receiving the following error: java.lang.OutOfMemoryError: Java heap ...
Enes's user avatar
  • 69
0 votes
1 answer
66 views

How to drop all NA columns in a SparkDataFrame with SparkR?

Once again, I'm facing a problem that I can't transcribe under SparkR. I have a SparkDataFrame which some columns contain only NAs, and I want to delete all these columns. I discovered SparkR recently,...
Discus23's user avatar
  • 511
0 votes
2 answers
100 views

How to add multiple calculated columns to a SparkDataFrame using SparkR?

Now I'm stuck on a rather basic case, but I can't find a clever solution with SparkR ... From N columns in my SparkDataFrame, I need to create N new calculated columns. df <- data.frame(V1 = base::...
Discus23's user avatar
  • 511
1 vote
1 answer
73 views

sparklyr: fill `spark_read_parquet path` argument from list

I'd like to process several files in the same s3 bucket in the same way. So, I created a list of the filenames: dt <- seq(as.Date("1991/12/23"), by = "day", length.out = 5) ...
HJT's user avatar
  • 317
0 votes
1 answer
149 views

How to use an undefined list of variables as column names in a SparkDataFrame with SparkR?

I keep progressing in the world of SparkR, and I am now facing an issue which I am unable to resolve. Working on SparkDataFrame manipulations, I may want to update some columns, or aggregate others. I ...
Discus23's user avatar
  • 511
0 votes
1 answer
239 views

Apply a function to multiple columns of a SparkDataFrame, at once

In basic R, this is how I apply a function to multiple columns at once, using data.table : d <- data.table(V1 = rep(1:2, 4:3), V2 = c(1, 2, 4, 5, 2, 3, 4), V3 = 1:7, V4 = sample(letters, 7)) Cols &...
Discus23's user avatar
  • 511
2 votes
2 answers
178 views

How to store SparkR result into an R object?

Still new to the world of Azure Databricks, the use of SparkR remains very obscure to me, even for very simple tasks... It took me a very long time to find how to count distinct values, and I'm not ...
Discus23's user avatar
  • 511

1
2 3 4 5
16