Questions tagged [sparklyr]
sparklyr is an alternative R interface for Apache Spark
776
questions
1
vote
0
answers
15
views
Calculate MAD in spark
calcola_mad <- function(dati) {
mediana_calcolata <- quantile(dati, 0.5)
deviazioni_assolute <- abs(dati - mediana_calcolata)
mad_calcolata <- mean(deviazioni_assolute)
return(...
0
votes
0
answers
26
views
Connect to bigquery from databricks using sparklyr
I'm pretty new to databricks. I have it running with Gloogle Cloud.
I have been able to connect to bigquery where I have my tables, and bring some data using SparkR:
library(SparkR)
sparkR.session()
...
1
vote
0
answers
76
views
How can you write to a Databricks catalog from R
I am currently trying to connect to databricks from a remote RStudio server.
Using the python package databricks-connect and sparklyr I am able to connect and read in delta tables from a catalog on ...
0
votes
1
answer
66
views
Add leading zeros to character column in a Spark data frame using sparklyr
How can I add leading zeroes to data$var1 in the following MWE?
library(sparklyr)
data <- data.frame(
var1 = c("ab", "abc", "abcd"),
var2 = c(1, 2, 3)
)
data <- ...
1
vote
0
answers
70
views
Why is a line break being added to my generated SQL and breaking my script?
I created an R environment on Conda 23.1 and below is my sessionInfo:
R version 4.2.2 (2022-10-31)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.9 (Maipo)...
0
votes
1
answer
84
views
What is the tidyr pivot_wider equivalent to the index argument in Python's pivot_table?
In this question (What is the best way in python to create a confusion matrix out of an already aggregated table with counts of actual labels and predicted labels), it is relatively straightforward to ...
1
vote
0
answers
26
views
Parallelization of ml_linear_regression in sparklyr
I am working with a dataframe that has the following structure, using sparklyr and dplyr:
lab | x | y
------|---|---
a | 1 | 1
a | 2 | 2
a | 3 | 3
b | 1 | 2
b | 2 | 4
b | 3 ...
0
votes
3
answers
111
views
Spark-Cassandra data retrieval error, although I can describe columns: java.lang.NoClassDefFoundError
(first StackOverflow post - I've searched but may have overseen an answer - if so, I appologize...)
I have a local Spark standalone instance running, and have tested it quite thoroughly using Sparklyr ...
0
votes
1
answer
80
views
Sparklyr / Dplyr Solution for counting occurrences of each value in all columns to one table
I would like to get a table from my data where the first column has a list of values that show up in the data and each additional column corresponds to a column in the data and gives the number of ...
0
votes
0
answers
110
views
How to process data larger than memory in R locally?
I have 83GB of data in the shape of CSVs stored in AWS S3. Usually I would use a Spark implementation in R (sparklyr) in AWS EMR to process the job granting enough memory.
However, I am not able to ...
0
votes
0
answers
126
views
Running sparklyr with YARN Docker-mode in EMR cluster
I am trying to set up an sparklyr job in EMR. I am using YARN Docker-mode (YARN_CONTAINER_RUNTIME_TYPE=docker) and deploying the job with spark-submit:
sudo spark-submit --master yarn --deploy-mode ...
10
votes
3
answers
473
views
R and sparklyr: Why is a simple query so slow?
This is my code. I run it in databricks.
library(sparklyr)
library(dplyr)
library(arrow)
sc <- spark_connect(method = "databricks")
tbl_change_db(sc, "prod")
trip_ids <- ...
1
vote
0
answers
22
views
sparklyr: Permission denied
I am having problems stablishing connection with spark_connect(master = "local") it has this error:
Error in file(con, "r") : cannot open the connection
In addition: Warning ...
0
votes
1
answer
212
views
How to read using sparklyr csv files located in blob storage without downloading it?
I´m using the following credentials auth for logging in blob storage in R:
library(AzureStor)
account_endpoint <- "https://mycorporation.blob.core.windows.net"
account_key <- &...
0
votes
1
answer
97
views
How to convert RDD to spark dataframe using sparklyr?
I have a lot of files with text data pushed by azure IOT on a blob storage in a lot of folders, and I want to read them and have a delta lake table with one row for each line of a file. I used to read ...
0
votes
0
answers
16
views
Row identifier in sparklyr ml_kmeans
I was looking for a way to keep a row identifier attached to the output (specifically the cluster assignment) when performing cluster analysis with the sparklyr ml_kmeans function.
I have been simply
...
0
votes
0
answers
8
views
Sparklyr spark_connect error in prepare_windows_environment: FindFileOwnerAndPermission error (1789) [duplicate]
Attempting to set up a local spark connection in R with spark_connect fails.
sc <- sparklyr::spark_connect(master = "local")
The error message:
Error in prepare_windows_environment(...
2
votes
1
answer
47
views
Remove duplicate numbers separated by a symbol in a string using Hive's REGEXP_REPLACE
I have a spark dataframe with a string column that includes numbers separated by ;, for example: 862;1595;17;862;49;862;19;100;17;49, I would like to remove the duplicated numbers, leaving the ...
0
votes
1
answer
101
views
group by and concatenate string using sparklyr
There are a number of questions asking precisely the same thing but none within the context of a sparklyr environment. How does one group by a column and then concatenate the values of some other ...
2
votes
1
answer
538
views
Connecting to Azure Databricks from R using jdbc and sparklyr
I'm trying to connect my on-premise R environment to an Azure Databricks backend using sparklyr and jdbc. I need to perform operations in databricks and then collect the results locally. Some ...
2
votes
0
answers
36
views
Sparklyr problem with tidy() function in context of ml_generalized_linear_regression() compared to ml_logistic_regression()
I am trying to build a model through sparklyr.
I started with exploratory analysis and then data cleaning (one hot encoding mostly).
I then built a logistic regression model with a binary outcome ...
0
votes
1
answer
121
views
Sparklyr spark_apply function on equal groups to run efficiently
How do I run a custom function efficiently in chunks within sparklyr environment?
I have a haversine function to calculate distance between 2 sets of Lat long within 1 data frame. As you can imagine, ...
1
vote
0
answers
51
views
Pivot_wider is not working as expected with spark_apply in R
Pivot_wider is not working as expected with spark_apply
[https://github.com/sparklyr/sparklyr/issues/3304]
d <- data. Frame(
c1 = c(11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12)
, ...
1
vote
0
answers
117
views
Using sparklyr and geospark in R, convert Spark DataFrame to Spark Spatial DataFrame (for point to line distance calculation)
I have a Spark DataFrame with latitude and longitude where I'm trying to calculate the distance between the coordinates and a polyline.
The dataframe I'm working with is large (about 10 billion ...
0
votes
1
answer
90
views
Sparklyr slice_max(with_ties = FALSE) returning extra rows
I am using sparklyr on databricks to query some delta tables.
I expect the below code to return 5 rows, but it returns all rows, as if slice_max is only doing arrange() and no slicing.
db %>%
...
2
votes
0
answers
657
views
sparklyr multiple Databricks notebooks, second connections fails with org.apache.spark.SparkException: Missing Credential Scope
First Databricks notebook
After restart of the cluster this is working fine:
library(sparklyr)
sc <- sparklyr::spark_connect(method="databricks")
iris_tbl <- copy_to(sc, iris)
Second ...
2
votes
1
answer
205
views
Use multiple spark connections in a databricks notebook
I want to connect to two database tables in a single databricks (R) notebook using spark and sparklyr:
sc_one <- spark_connect(method="databricks")
tbl_change_db(sc_one, "...
0
votes
1
answer
262
views
Using sparklyr write SQL on new Spark Dataframe
I am able to create a Spark dataframe using:
df_01 <- sdf_sql(sc, "select * from database.table")
How can I use sparklyr to access the created Spark dataframe df_01 in a subsequent SQL ...
0
votes
2
answers
342
views
Hive tables not showing in Spark session
If I run DBI::dbGetQuery(sc, "SHOW DATABASES") in R, I get as result only default database.
And not the full list of hive tables created from the hive> command line...
Also in the R ...
0
votes
1
answer
63
views
Sparklyr/Spark NLP connect via YARN
I'm new to sparklyr and spark nlp. Had gotten a local connection running no problem and test data was saving and being read back etc. Today when I loaded the real data which is a batch of text data ...
2
votes
0
answers
735
views
Warning "ORDER BY is ignored in subqueries without LIMIT" in R sparklyr
Issue
Sparklyr script output is filled with the same warning message (over and over again) :
"ORDER BY is ignored in subqueries without LIMIT
Do you need to move arrange() later in the pipeline ...
1
vote
1
answer
162
views
Delete delta table in Databricks with R
I'm trying to delete a delta table using R from Databricks.
It works if I do :
system("rm -rf /my_dir/my_table")
But I don't really like this solution...
Is there a more elegant solution in ...
0
votes
1
answer
170
views
R: Create group id in sparklyr dataframe with varying levels of aggregation
I have a sparklyr dataframe similar to the one generated with the code below.
# Load libraries
library(sparklyr)
library(dplyr)
library(magrittr)
# Connect to spark
sc <- spark_connect(master = &...
0
votes
1
answer
82
views
Running sparklyr Functions in Parallel
I am currently using sparklyr to transform a large dataset and calculate a set of metrics based on an input start and end date. This function is run on the same dataframe for multiple start and end ...
0
votes
0
answers
145
views
How do I register custom JDBC dialect in Rstudio?
I'm trying to analyze bigquery data in Rstudio server running on a google dataproc cluster. However, due to the memory limitations of Rstudio, I intend to run queries on this data in sparklyr but I ...
0
votes
1
answer
351
views
Connecting MongoDB to Spark and sparklyr
I'm trying to assemble a big data infrastructure on my local machine using MongoDB > Apache Spark > RStudio sparklyr. I can't find a solution to connect sparklyr to MongoDB. There are a small ...
0
votes
0
answers
250
views
Alternative to slice() function in sparklyr
Currently I'm using this code in order to subset unique rows grouped by Loan_ID where Execution_Date is the closest to predetermined date (in my case "2022-03-31")
The example is as follows:
...
1
vote
0
answers
67
views
Can't escape back tick from string error when bind rows in spark data frame
When running sparklyr::sdf_bind_rows on spark dataframes in R environment I get this error:
Can't escape back tick from string.
I made sure to remove any '`' from the tables.
Sometimes when I perform ...
0
votes
1
answer
103
views
Error reading data into Spark using spraklyr::spark_read_csv
I'm running Spark in 'standalone' mode on a local machine in Docker containers. I have a master and two workers, each is running in its own Docker container. In each of the containers the path /opt/...
0
votes
0
answers
51
views
Sparklyr/sparc_read_csv: Ignore \r\n while reading data
I have received a large CSV file (50 million entries) with the following format:
url,httpsStatusCode,newURL,Payload
where the last variable includes the complete HTML code. vroom() is able to read ...
0
votes
1
answer
77
views
Find size of intersection of two list-columns in sparklyr
I am working with a tbl_spark in sparklyr.
I have a spark Dataframe with two list-type columns, and I would like to output two things:
The intersection of both lists (as a list)
The number of ...
0
votes
1
answer
288
views
How to extend/complete the dates in the Date column of a table? Using dplyr/sparklyr
I have this data with missing dates in the Date column, how can I expand this column to the full range of dates since the start and end of each grouping?
The grouping are the columns Material and ...
2
votes
0
answers
136
views
Tidymodels worfklow_map() function not working on Spark cluster with R sparklyr
I am attempting to run a time series cross validation ML tuning process on a Spark cluster (sparklyr on Databricks), but am getting an error. The packages I'm using are tidymodels with modeltime. The ...
1
vote
1
answer
162
views
What is the issue with my pagerank algorithm code?
df1 \<- tbl(sc ,"databricks.tablename")
g \<- graph_from_data_frame(df1, directed = TRUE, vertices = NULL) # Directed Igraph creation
is_weighted(g) #Check for Weight
pagerank \<-...
0
votes
0
answers
122
views
Java Heap Space Error in SparklyR & Tidymodels
Hi,
I am trying to model a large dataset (2000000+ rows, ~150 columns) using the tidymodels and the spark engine. However, I am receiving the following error: java.lang.OutOfMemoryError: Java heap ...
1
vote
1
answer
165
views
Cannot connect to Spark through sparklyr in RStudio
I connect to spark using this code:
sc <- spark_connect(master = "local")
Got some error messages:
Exception in thread "main" java.lang.ExceptionInInitializerError
at org....
0
votes
1
answer
94
views
Error when creating H2O context using RSparkling
I am running Spark 2.4.4 using Yarn and interfacing using RSparkling and Sparklyr
As per these instructions I've
Installed Sparklyr
Called the library for Sparklyr
Removed any prior installs of H2O
...
0
votes
1
answer
124
views
Trouble getting H2O to work with Sparklyr
I am trying to get H2O working with Sparklyr on my spark cluster (yarn)
spark_version(sc) = 2.4.4
My spark cluster is running V2.4.4
According to this page the compatible version with my spark is 2.4....
1
vote
1
answer
73
views
sparklyr: fill `spark_read_parquet path` argument from list
I'd like to process several files in the same s3 bucket in the same way. So, I created a list of the filenames:
dt <- seq(as.Date("1991/12/23"), by = "day", length.out = 5)
...
3
votes
0
answers
333
views
How to display Sparklyr table in a clean readable format similar to the output of display() in Databricks?
There exist a Databricks’s built-in display() function (see documentation here) which allow users to display R or SparkR dataframe in a clean and human readable manner where user can scroll to see all ...