Questions tagged [sparklyr]

sparklyr is an alternative R interface for Apache Spark

Filter by
Sorted by
Tagged with
1 vote
0 answers
15 views

Calculate MAD in spark

calcola_mad <- function(dati) { mediana_calcolata <- quantile(dati, 0.5) deviazioni_assolute <- abs(dati - mediana_calcolata) mad_calcolata <- mean(deviazioni_assolute) return(...
LAURA's user avatar
  • 11
0 votes
0 answers
26 views

Connect to bigquery from databricks using sparklyr

I'm pretty new to databricks. I have it running with Gloogle Cloud. I have been able to connect to bigquery where I have my tables, and bring some data using SparkR: library(SparkR) sparkR.session() ...
filemonPi's user avatar
1 vote
0 answers
76 views

How can you write to a Databricks catalog from R

I am currently trying to connect to databricks from a remote RStudio server. Using the python package databricks-connect and sparklyr I am able to connect and read in delta tables from a catalog on ...
Hejlesen's user avatar
0 votes
1 answer
66 views

Add leading zeros to character column in a Spark data frame using sparklyr

How can I add leading zeroes to data$var1 in the following MWE? library(sparklyr) data <- data.frame( var1 = c("ab", "abc", "abcd"), var2 = c(1, 2, 3) ) data <- ...
Christoffer's user avatar
1 vote
0 answers
70 views

Why is a line break being added to my generated SQL and breaking my script?

I created an R environment on Conda 23.1 and below is my sessionInfo: R version 4.2.2 (2022-10-31) Platform: x86_64-conda-linux-gnu (64-bit) Running under: Red Hat Enterprise Linux Server 7.9 (Maipo)...
MyNameHere's user avatar
0 votes
1 answer
84 views

What is the tidyr pivot_wider equivalent to the index argument in Python's pivot_table?

In this question (What is the best way in python to create a confusion matrix out of an already aggregated table with counts of actual labels and predicted labels), it is relatively straightforward to ...
Alpha Bravo's user avatar
1 vote
0 answers
26 views

Parallelization of ml_linear_regression in sparklyr

I am working with a dataframe that has the following structure, using sparklyr and dplyr: lab | x | y ------|---|--- a | 1 | 1 a | 2 | 2 a | 3 | 3 b | 1 | 2 b | 2 | 4 b | 3 ...
Giacomo Marcon's user avatar
0 votes
3 answers
111 views

Spark-Cassandra data retrieval error, although I can describe columns: java.lang.NoClassDefFoundError

(first StackOverflow post - I've searched but may have overseen an answer - if so, I appologize...) I have a local Spark standalone instance running, and have tested it quite thoroughly using Sparklyr ...
James75's user avatar
0 votes
1 answer
80 views

Sparklyr / Dplyr Solution for counting occurrences of each value in all columns to one table

I would like to get a table from my data where the first column has a list of values that show up in the data and each additional column corresponds to a column in the data and gives the number of ...
34jbonz's user avatar
  • 118
0 votes
0 answers
110 views

How to process data larger than memory in R locally?

I have 83GB of data in the shape of CSVs stored in AWS S3. Usually I would use a Spark implementation in R (sparklyr) in AWS EMR to process the job granting enough memory. However, I am not able to ...
Alberto Agudo Dominguez's user avatar
0 votes
0 answers
126 views

Running sparklyr with YARN Docker-mode in EMR cluster

I am trying to set up an sparklyr job in EMR. I am using YARN Docker-mode (YARN_CONTAINER_RUNTIME_TYPE=docker) and deploying the job with spark-submit: sudo spark-submit --master yarn --deploy-mode ...
pvieito's user avatar
  • 117
10 votes
3 answers
473 views

R and sparklyr: Why is a simple query so slow?

This is my code. I run it in databricks. library(sparklyr) library(dplyr) library(arrow) sc <- spark_connect(method = "databricks") tbl_change_db(sc, "prod") trip_ids <- ...
Funkwecker's user avatar
1 vote
0 answers
22 views

sparklyr: Permission denied

I am having problems stablishing connection with spark_connect(master = "local") it has this error: Error in file(con, "r") : cannot open the connection In addition: Warning ...
Joshua Cervantes Artavia's user avatar
0 votes
1 answer
212 views

How to read using sparklyr csv files located in blob storage without downloading it?

I´m using the following credentials auth for logging in blob storage in R: library(AzureStor) account_endpoint <- "https://mycorporation.blob.core.windows.net" account_key <- &...
Rodrigo H. Ozon's user avatar
0 votes
1 answer
97 views

How to convert RDD to spark dataframe using sparklyr?

I have a lot of files with text data pushed by azure IOT on a blob storage in a lot of folders, and I want to read them and have a delta lake table with one row for each line of a file. I used to read ...
robin loche's user avatar
0 votes
0 answers
16 views

Row identifier in sparklyr ml_kmeans

I was looking for a way to keep a row identifier attached to the output (specifically the cluster assignment) when performing cluster analysis with the sparklyr ml_kmeans function. I have been simply ...
retzerjj's user avatar
0 votes
0 answers
8 views

Sparklyr spark_connect error in prepare_windows_environment: FindFileOwnerAndPermission error (1789) [duplicate]

Attempting to set up a local spark connection in R with spark_connect fails. sc <- sparklyr::spark_connect(master = "local") The error message: Error in prepare_windows_environment(...
Alex W's user avatar
  • 509
2 votes
1 answer
47 views

Remove duplicate numbers separated by a symbol in a string using Hive's REGEXP_REPLACE

I have a spark dataframe with a string column that includes numbers separated by ;, for example: 862;1595;17;862;49;862;19;100;17;49, I would like to remove the duplicated numbers, leaving the ...
Cyrus Mohammadian's user avatar
0 votes
1 answer
101 views

group by and concatenate string using sparklyr

There are a number of questions asking precisely the same thing but none within the context of a sparklyr environment. How does one group by a column and then concatenate the values of some other ...
Cyrus Mohammadian's user avatar
2 votes
1 answer
538 views

Connecting to Azure Databricks from R using jdbc and sparklyr

I'm trying to connect my on-premise R environment to an Azure Databricks backend using sparklyr and jdbc. I need to perform operations in databricks and then collect the results locally. Some ...
Leandro R's user avatar
2 votes
0 answers
36 views

Sparklyr problem with tidy() function in context of ml_generalized_linear_regression() compared to ml_logistic_regression()

I am trying to build a model through sparklyr. I started with exploratory analysis and then data cleaning (one hot encoding mostly). I then built a logistic regression model with a binary outcome ...
RoSull92's user avatar
0 votes
1 answer
121 views

Sparklyr spark_apply function on equal groups to run efficiently

How do I run a custom function efficiently in chunks within sparklyr environment? I have a haversine function to calculate distance between 2 sets of Lat long within 1 data frame. As you can imagine, ...
Choc_waffles's user avatar
1 vote
0 answers
51 views

Pivot_wider is not working as expected with spark_apply in R

Pivot_wider is not working as expected with spark_apply [https://github.com/sparklyr/sparklyr/issues/3304] d <- data. Frame( c1 = c(11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12) , ...
user1986858's user avatar
1 vote
0 answers
117 views

Using sparklyr and geospark in R, convert Spark DataFrame to Spark Spatial DataFrame (for point to line distance calculation)

I have a Spark DataFrame with latitude and longitude where I'm trying to calculate the distance between the coordinates and a polyline. The dataframe I'm working with is large (about 10 billion ...
Rob Marty's user avatar
  • 398
0 votes
1 answer
90 views

Sparklyr slice_max(with_ties = FALSE) returning extra rows

I am using sparklyr on databricks to query some delta tables. I expect the below code to return 5 rows, but it returns all rows, as if slice_max is only doing arrange() and no slicing. db %>% ...
Pickle's user avatar
  • 74
2 votes
0 answers
657 views

sparklyr multiple Databricks notebooks, second connections fails with org.apache.spark.SparkException: Missing Credential Scope

First Databricks notebook After restart of the cluster this is working fine: library(sparklyr) sc <- sparklyr::spark_connect(method="databricks") iris_tbl <- copy_to(sc, iris) Second ...
ran's user avatar
  • 21
2 votes
1 answer
205 views

Use multiple spark connections in a databricks notebook

I want to connect to two database tables in a single databricks (R) notebook using spark and sparklyr: sc_one <- spark_connect(method="databricks") tbl_change_db(sc_one, "...
Sanchez333's user avatar
0 votes
1 answer
262 views

Using sparklyr write SQL on new Spark Dataframe

I am able to create a Spark dataframe using: df_01 <- sdf_sql(sc, "select * from database.table") How can I use sparklyr to access the created Spark dataframe df_01 in a subsequent SQL ...
Scott's user avatar
  • 446
0 votes
2 answers
342 views

Hive tables not showing in Spark session

If I run DBI::dbGetQuery(sc, "SHOW DATABASES") in R, I get as result only default database. And not the full list of hive tables created from the hive> command line... Also in the R ...
Faneron's user avatar
0 votes
1 answer
63 views

Sparklyr/Spark NLP connect via YARN

I'm new to sparklyr and spark nlp. Had gotten a local connection running no problem and test data was saving and being read back etc. Today when I loaded the real data which is a batch of text data ...
Magnetar's user avatar
  • 105
2 votes
0 answers
735 views

Warning "ORDER BY is ignored in subqueries without LIMIT" in R sparklyr

Issue Sparklyr script output is filled with the same warning message (over and over again) : "ORDER BY is ignored in subqueries without LIMIT Do you need to move arrange() later in the pipeline ...
TOMC's user avatar
  • 103
1 vote
1 answer
162 views

Delete delta table in Databricks with R

I'm trying to delete a delta table using R from Databricks. It works if I do : system("rm -rf /my_dir/my_table") But I don't really like this solution... Is there a more elegant solution in ...
TOMC's user avatar
  • 103
0 votes
1 answer
170 views

R: Create group id in sparklyr dataframe with varying levels of aggregation

I have a sparklyr dataframe similar to the one generated with the code below. # Load libraries library(sparklyr) library(dplyr) library(magrittr) # Connect to spark sc <- spark_connect(master = &...
Adriana LE's user avatar
0 votes
1 answer
82 views

Running sparklyr Functions in Parallel

I am currently using sparklyr to transform a large dataset and calculate a set of metrics based on an input start and end date. This function is run on the same dataframe for multiple start and end ...
Dave Wilson's user avatar
0 votes
0 answers
145 views

How do I register custom JDBC dialect in Rstudio?

I'm trying to analyze bigquery data in Rstudio server running on a google dataproc cluster. However, due to the memory limitations of Rstudio, I intend to run queries on this data in sparklyr but I ...
Gboyega.A's user avatar
0 votes
1 answer
351 views

Connecting MongoDB to Spark and sparklyr

I'm trying to assemble a big data infrastructure on my local machine using MongoDB > Apache Spark > RStudio sparklyr. I can't find a solution to connect sparklyr to MongoDB. There are a small ...
Willian Adamczyk's user avatar
0 votes
0 answers
250 views

Alternative to slice() function in sparklyr

Currently I'm using this code in order to subset unique rows grouped by Loan_ID where Execution_Date is the closest to predetermined date (in my case "2022-03-31") The example is as follows: ...
mabramov's user avatar
  • 161
1 vote
0 answers
67 views

Can't escape back tick from string error when bind rows in spark data frame

When running sparklyr::sdf_bind_rows on spark dataframes in R environment I get this error: Can't escape back tick from string. I made sure to remove any '`' from the tables. Sometimes when I perform ...
tzviya's user avatar
  • 538
0 votes
1 answer
103 views

Error reading data into Spark using spraklyr::spark_read_csv

I'm running Spark in 'standalone' mode on a local machine in Docker containers. I have a master and two workers, each is running in its own Docker container. In each of the containers the path /opt/...
amitr's user avatar
  • 800
0 votes
0 answers
51 views

Sparklyr/sparc_read_csv: Ignore \r\n while reading data

I have received a large CSV file (50 million entries) with the following format: url,httpsStatusCode,newURL,Payload where the last variable includes the complete HTML code. vroom() is able to read ...
Tom's user avatar
  • 329
0 votes
1 answer
77 views

Find size of intersection of two list-columns in sparklyr

I am working with a tbl_spark in sparklyr. I have a spark Dataframe with two list-type columns, and I would like to output two things: The intersection of both lists (as a list) The number of ...
MRipley's user avatar
0 votes
1 answer
288 views

How to extend/complete the dates in the Date column of a table? Using dplyr/sparklyr

I have this data with missing dates in the Date column, how can I expand this column to the full range of dates since the start and end of each grouping? The grouping are the columns Material and ...
SantosDev's user avatar
2 votes
0 answers
136 views

Tidymodels worfklow_map() function not working on Spark cluster with R sparklyr

I am attempting to run a time series cross validation ML tuning process on a Spark cluster (sparklyr on Databricks), but am getting an error. The packages I'm using are tidymodels with modeltime. The ...
skklogw7's user avatar
1 vote
1 answer
162 views

What is the issue with my pagerank algorithm code?

df1 \<- tbl(sc ,"databricks.tablename") g \<- graph_from_data_frame(df1, directed = TRUE, vertices = NULL) # Directed Igraph creation is_weighted(g) #Check for Weight pagerank \<-...
Anand Salimath's user avatar
0 votes
0 answers
122 views

Java Heap Space Error in SparklyR & Tidymodels

Hi, I am trying to model a large dataset (2000000+ rows, ~150 columns) using the tidymodels and the spark engine. However, I am receiving the following error: java.lang.OutOfMemoryError: Java heap ...
Enes's user avatar
  • 69
1 vote
1 answer
165 views

Cannot connect to Spark through sparklyr in RStudio

I connect to spark using this code: sc <- spark_connect(master = "local") Got some error messages: Exception in thread "main" java.lang.ExceptionInInitializerError at org....
Jinghong Zeng's user avatar
0 votes
1 answer
94 views

Error when creating H2O context using RSparkling

I am running Spark 2.4.4 using Yarn and interfacing using RSparkling and Sparklyr As per these instructions I've Installed Sparklyr Called the library for Sparklyr Removed any prior installs of H2O ...
TheGoat's user avatar
  • 2,625
0 votes
1 answer
124 views

Trouble getting H2O to work with Sparklyr

I am trying to get H2O working with Sparklyr on my spark cluster (yarn) spark_version(sc) = 2.4.4 My spark cluster is running V2.4.4 According to this page the compatible version with my spark is 2.4....
TheGoat's user avatar
  • 2,625
1 vote
1 answer
73 views

sparklyr: fill `spark_read_parquet path` argument from list

I'd like to process several files in the same s3 bucket in the same way. So, I created a list of the filenames: dt <- seq(as.Date("1991/12/23"), by = "day", length.out = 5) ...
HJT's user avatar
  • 317
3 votes
0 answers
333 views

How to display Sparklyr table in a clean readable format similar to the output of display() in Databricks?

There exist a Databricks’s built-in display() function (see documentation here) which allow users to display R or SparkR dataframe in a clean and human readable manner where user can scroll to see all ...
SG_'s user avatar
  • 69

1
2 3 4 5
16