Questions tagged [spark-java]
Spark is a lightweight web application framework inspired by Ruby's Sinatra. DO NOT USE FOR QUESTIONS ABOUT APACHE SPARK.
544
questions
0
votes
1
answer
39
views
Apache Spark , detect key change on a row and group rows
I have the below dataframe in spark where I need to detect the key change ( on column rec) and create a new column called groupId. For example the first row and second row belong to one group until ...
0
votes
0
answers
15
views
Spark java IllegalArgumentException: Output column features already exists but cant find it
im running a linear regression model using spark java
// assembling data
featureAssembler = new VectorAssembler().setInputCols(new String[]{"genre","sector"})....
0
votes
0
answers
20
views
How do I set an expire time on the session ID cookie (JSESSIONID) in Spark Framework (SparkJava)?
I have a question.
I would like to set an expire time on a Session.
I am creating a Web API in Java8 and Spark Framework(2.7.2).
I am using Session to keep the authentication state.
I know that ...
0
votes
0
answers
11
views
How can I generate org.apache.spark.rpc.RpcTimeoutException during Spark Streaming
I'm very new to spark and trying to test my spark streaming application for org.apache.spark.rpc.RpcTimeoutException.
How can I simulate a scenario where this exception is thrown for a streaming query....
0
votes
1
answer
28
views
What is the substitute of boolean true in spark sql case when statement?
I'm using spark in Java and I'm facing a very weird problem. I would like to know the substitute of boolean true in spark sql.
In oracle we can use something like -
CASE WHEN 1=1 THEN 'Y' ELSE 'N' END ...
3
votes
1
answer
36
views
Request body gets lost in http get request. (Java-Spark)
When I sent the following request to my program, the body gets lost:
GET http://localhost:4567/contacts/get-all
Authorization : test
Content-Type : text/plain
Hello
Java Code
public class Test {
...
0
votes
0
answers
21
views
Join/Merge two Dataset<Row> in spark java
I am simply trying to concatenate two different Datasets (Dataset<Row>) using spark in Java.
The problem I am having is the datasets are going to have different number of columns and possibly ...
0
votes
0
answers
16
views
Predict on user input data with Spark ML Pipeline in Java
I have made a model using Spark ML in Java with a dataset that was split into train and test. I was wondering how to use the model on real time data such as from a user input ?
Tokenizer tokenizer ...
0
votes
0
answers
10
views
How we can write spring interceptor inside sparkjava?
How we can writer spring interceptor with sparkjava?
like https://howtodoinjava.com/spring-core/spring-mvc-interceptor-example/
0
votes
0
answers
17
views
sparkjava shutsdown randomly
We have an REST application developed in sparkjava.
Suddenly at some point it shutting down without any cause in the log.
July 1, 2022 17:43:27.087 [Thread-27] INFO spark.embeddedserver.jetty....
0
votes
0
answers
38
views
Convert String into Spark Expression and evaluate it
We want to evaluate expression which will be in java String as spark Expression without Dataframe.
for example, these are working
import org.apache.spark.sql.catalyst.InternalRow;
import ...
0
votes
0
answers
19
views
sparkjava : Secure one of the external end point but getting throwBeforeRouteMappingException
I am new to sparkJava and I am trying to create multiple endpoints. I have two endpoints, the first one is internal and the second one is external.
What I learned from here is this is I have to ignite ...
0
votes
0
answers
7
views
Why does SparkJava not deploy with version 2.9.3?
I have an issue trying to deploy a sparkjava web app using 2.9.3. The error I get is "java.lang.IllegalStateException: Unable to find ServletContextHandler for provided ServletContext."
If I ...
0
votes
1
answer
125
views
How to create a struct column from a list of column names in Spark with Java?
I have a DataFrame with multiple columns, e.g.
root
|-- playerName
|-- country
|-- bowlingAvg
|-- bowlingSR
|-- wickets
|-- battingAvg
|-- battingSR
|-- runs
I also have a list of the column ...
-1
votes
1
answer
41
views
My SparkJava resource server gets 403 errors when trying to validate access tokens
I want to set up a very basic REST API using Spark-java, which just checks an access token obtained from my own authorisation server. It creates a GET request to the authorisation server's /oauth/...
0
votes
0
answers
45
views
how to convert form data request.body in SPARK to map of key and value
----------------------------170486398443765912081605
Content-Disposition: form-data; name="id"
11
I want to remove all the dashes, numbers, and content disposition and the output be a map -...
0
votes
0
answers
29
views
spark jdbc read for greenplum -how read a large table
the code as:
Dataset<Row> invoke = spark.read().format("jdbc").option("url", url)
.option("dbtable", query)
.option("partitionColumn", "id")
....
3
votes
1
answer
258
views
How to union two dataframes which have same number of columns?
Dataframe df1 contains columns : a, b, c, d, e (Empty dataframe)
Dataframe df2 contains columns : b, c, d, e, _c4 (Contains Data)
I want to do a union on these two dataframes. I tried using
df1.union(...
0
votes
0
answers
114
views
How to make sure a (liveness) endpoint, served by spark-java (jetty) will always find a thread available in order to respond quickly enough?
I've got a scala application with a few endpoints served by spark-java (with jetty).
That application is running on a kubernetes pod, and one of the endpoints takes a lot of time to respond (around 5 ...
1
vote
1
answer
67
views
How to create an edge list using a column of node path of ArrayType in Spark?
I have a Spark Dataset containing a single column of ArrayType which denotes the path from one user to another through their mutual friends
path
["Amy","John","Wally"]
...
0
votes
1
answer
35
views
Can't deploy SparkJava program on Heroku , getting Insufficient privileges error
I have a simple sparkJava endpoint, i am trying to deploy it on Heroku but i am getting the following error.
Failed to deploy application: Insufficient privileges to "spark-heroku-example" ...
0
votes
0
answers
30
views
spark-java 2.9.3 Jetty security issue
could you upgrade the jetty version in your last release?
jetty-webapp-9.4.31.v20200723.jar-Policy Violation - Security Severity - High
jetty-io-9.4.31.v20200723.jar - Policy Violation - Security ...
0
votes
0
answers
47
views
Spark conversion error on the executor JDBC SQLSERVER partition option
I'm first reading lower bound and upper bound using a:
select max(timestamp) ,min(timestamp) from table name
extracting
Row row=query.collectasList().get(0).getString(0) as lowerbound and upperbound
...
0
votes
0
answers
21
views
How to read data from mongo to sparkjava?
I have a problem when I read data from mongo to my sparkJava project.
I want to take all the data from rdd, convert it to "User" and save it in a users list.
I try to run over my rdds with ...
0
votes
1
answer
172
views
Creating empty Spark dataframe and adding arbitrary values
I have created an empty dataframe and then trying to add columns and values to the dataframe.
Dataset<Row> runMetadata = sparkSession.emptyDataFrame();
runMetadata = runMetadata.withColumn("...
2
votes
1
answer
312
views
FileNotFoundException (operation not permitted) error but signs point to file present?
I am trying to deploy a SparkJava REST app in a Kubernetes container on my Windows machine.
Windows 10
Kubernetes v1.22.5
(edit) base image: openjdk:8-alpine
I am trying to read in a properties file ...
2
votes
1
answer
79
views
Spark's DELETE handler returns NULL when accessing any request.queryParams key
I developed a very simple API using the Spark Framework (ref: sparkjava.com). I wrote a delete handler and tested it with Postman.
The Problem
When I try to assign a value using request.queryParams, ...
0
votes
0
answers
68
views
GET Method not fetching latest data i POSTED Java and Spark
So, this is the problem... when i insert new data to database using post method, it worked, the med table in database have new data. but when i fetch the data using GET Method, the data i inserted ...
0
votes
2
answers
235
views
Java Spark convert Empty values in Dataframe to null
I'm trying to convert all empty values in my spark Dataframe to null using:
df.withColumn(colname, when(df.col(colname).equalTo(""), null)
.otherwise(df....
0
votes
0
answers
18
views
Got NoClassDefFoundErrorerror when starting the server, might caused by standford.nlp in pom.xml
I got NoClassDefFoundErrorerror when trying to start my server. I guess the main problem is I have edu.standford.nlp library in my pom.xml. How should I solve it?
<dependency>
<...
1
vote
1
answer
305
views
IntelliJ cannot resolve symbol 'spark' when importing Spark Java
I am currently running the latest IntelliJ and I am trying to use SparkJava Webserver for a Gradle project I'm writing.
When I do compile and run with gradle everything works fine, however I am ...
0
votes
0
answers
33
views
Some of the Spark Jobs are not initializing and not getting the Control Back from the Spark Write Action
I have been using Spark 3.1.1 and Java driver and maintained the spark Cache for my data and created the view. Doing the below transformations.
Querying the data from the view
Doing the MySQL query ...
0
votes
1
answer
102
views
Spark Dataframe MapType filter on Keys and Values
I am using spark 2.4 & Java.
I am playing around with MapTypes in spark.
Consider the following code
public class MapTypes {
public static void main(String[] args) {
SparkSession spark ...
0
votes
0
answers
59
views
Call service method from spark
I'm using spark with spring boot. I have a service ( spring bean ), which I want to call for batch of data. The issue is that I keep getting Task not serializable exception.
javaRDD.foreachPartition(...
0
votes
2
answers
344
views
package spark does not exist
I'm just starting to learn Java.
I am using IntelliJ to create a SparkJava (see: sparkjava.com) app using openjdk 17. I followed the tutorial instructions outlined in "Setting up Spark with ...
1
vote
1
answer
206
views
Group by inside otherwise clause, spark java
I have this process in SparkJava (IntelliJ app) where I have a problem that I don`t know how to resolve yet. First I declare the dataset:
private static final String CONTRA1 = "contra1";
...
0
votes
0
answers
30
views
python array_split equivalent API in Spark Java to apply over a list
Basically array_split python functions splits the List in parts as per certain condition.
I need to implement array_split function in Spark Java. I was wondering if there is an existing inherent ...
1
vote
1
answer
243
views
Apache Spark 2.3.0 - How to merge two array<string> into one array<string>
I tried and googled but was not able to find the solution.
Merging two array<string> into one array<string> is possible and easy in 2.4.x versions but could not find a way to do it in 2.3....
0
votes
1
answer
534
views
Microsoft Active Directory SSO to docker container
I have a web application running in a docker container. I need to let only the users authenticated with Microsoft SSO (only the users in my AzureAD tenant). The web application is written in Spark ...
1
vote
1
answer
89
views
Firebase Auth REST - Log in with Microsoft Provider
I'm trying to log in users to a Java Web application written in Java Spark framework using firebase Auth REST API here. Reason is since I couldn't find a better way to integrate with Firebase from ...
1
vote
1
answer
649
views
How to call a REST API with Java Spark Framework?
I'm referring to the Spark micro framework here(not apache spark). I need to call a REST API, for instance, a google API, in an application written in Java Spark.
I don't see a direct way to do it in ...
0
votes
0
answers
71
views
SparkJava Websocket Server in Kotlin not working
At a glance, it may look like a duplicate, but I tried the answer(s) in the previous question(s). I have been trying to implement a websocket server using Spark Framework in kotlin. The server seems ...
0
votes
0
answers
87
views
Thread Local Ok?
I've implemented a simile web service use Spark Java
I have a question similar to what was asked in
Is it advisable to create a request context with ThreadLocal on the Spark framework
I was thinking ...
0
votes
2
answers
127
views
TimeZone conversion issue in Spark Java
I want to insert date in Central Timezone as creation_date. I changed the jvm timezone(US/Central) and created date with timezone(Timezone) and able to insert into DB in case of local master url but ...
1
vote
1
answer
110
views
What is wrong with my structure in Java project?
I got a Maven project setup with Spark. I followed Spark's documentation on how to create a simple get request: http://sparkjava.com/documentation#getting-started
but I get these errors:
What can be ...
1
vote
0
answers
42
views
Spark.unmap() is creating CORS issue while unmapping an API
It seems like the Spark.unmap(path) is creating the CORS issue. After using unmap, the request from the application running on localhost is not accepted by the SparkJava server, and it seems like its ...
1
vote
1
answer
114
views
How to query based on a list of DBRef field in Spark SQL
I have a mongo database that contains a list of DBRef objects. I am converting this collection into a data frame in spark.
This gives the below error. Does anyone know how to resolve this error ??
...
0
votes
1
answer
120
views
Postman and SparkJava response.body() aways null
My application was working fine before, but now my response.body() is always null and I don't know where the problem is.
this is my application main method(spark java framework)
post("/...
0
votes
1
answer
161
views
Spark java error, i cant start the application
i've using Spark framework, I just created the gradle project and added the spark dependency but when I start the application, it gives this error in console:
SLF4J: Failed to load class "org....
0
votes
1
answer
105
views
Select corresponding value of not included column in groupBy in spark dataframe in java [duplicate]
I have a dataframe as follows
col1, col2, version_time, col3
root
|-- col1: string (nullable = true)
|-- col2: integer (nullable = true)
|-- version_time: timestamp (nullable = true)
|-- col3: ...