Questions tagged [spark-java]
Spark is a lightweight web application framework inspired by Ruby's Sinatra. DO NOT USE FOR QUESTIONS ABOUT APACHE SPARK.
556
questions
0
votes
0
answers
19
views
Spark job throwing exceptions during loading data from Kafka to Hive
We have a big data cluster where we have data in Kakfa topics and we load it to hive using spark job (with java 8).I was using Cloudera 7.1.7 version and spark version (2.4.7.7.1.7.1000-141) SP1 and ...
-1
votes
1
answer
34
views
Spark Java sum is giving incorrect value
Spark Java sum is giving incorrect value
Java sample code is as below
List<Double> points = Arrays.asList(-6221.4, 6380.46);
Dataset<Row> dt = spark.createDataset(points, Encoders.DOUBLE()...
0
votes
0
answers
32
views
Unable to serve a static web page using Spark java framework
I have the below code - which is configured to serve static web page within the package (src/main/resources/public)
Which is not working and i dont see any exception in the logs as well
public void ...
0
votes
0
answers
20
views
Redirects to another site
Can I make redirect from current page to google.com (for example)?
I tried to use method redirect and header with Location.
Is it possible to somehow implement a redirect to another website, or only ...
1
vote
1
answer
63
views
How to set schema into spark.sql.function.from_csv?
I use spark-3.4.1-hadoop3 on windows 11. And I try to generate the schema to pass into from_csv function parameter.
Belows are my codes.
import org.apache.spark.sql.Column;
import org.apache.spark.sql....
0
votes
1
answer
43
views
repartition not working with xml file in Spark
I have dataframe which I want to save as multiple xml files. This is my code:
employees
.repartition(col("first_name"))
.write()
.option(&...
0
votes
1
answer
61
views
How to get index/position of column in dataframe (Spark sql Java)
I am using Spark Java (not scala, python).
I have to change my code so that my spark query will select all columns rather than a specific set of columns. (Like using select *). Before when I had a ...
0
votes
2
answers
42
views
Select a list of set columns AND other columns depending on their name/regex (Spark SQL Java)
I have a list of columns that I need to select. I have the field names for each column in this list, so it is easy to select.
public Column[] getSelectColumns()
{
return new Column[]{
...
0
votes
0
answers
66
views
Dynamically selecting column based on if that column exists in another column's value (Java Spark SQL)
I am using Spark SQL Java (Not scala or python). Here is a simplified example of my dataset (it has many more columns than this):
ID
References
E_id
F_id
G_id
0
[E_id]
0000
1
[E_id, F_id, G_id]
...
0
votes
1
answer
52
views
How to get attribute value with correct datatype from Row in java spark sql
I am using Spark Java (not scala or python).
I have a dataframe (Dataset<Row>) and I want to access specific fields from the Row. I have been able to get the field value if it is a String, but I ...
0
votes
0
answers
14
views
Is it possible to have executors evaluate part of the query instead of master finding the entire result set first in spark java?
I have 2 tables in postgres db that I would need to join and send the resulting output to kafka. To do that I wrote a java code that uses spark framework. The sample of my code is like so:
Main class:
...
0
votes
2
answers
38
views
What is the correct way to use Regex_replace to replace a '|' with a '-' in Spark Java's column named details?
I have a column named details which has special character as | I want to replace it with - . How can I do it in spark Java?
I have tried:
regex_replace(details ,"|","-")
...
0
votes
0
answers
59
views
Spark Error org.apache.spark.SparkException: Error communicating with MapOutputTracker during Spark submit
Getting below exception
org.apache.spark.SparkException: Error communicating with MapOutputTracker
at org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:523)
at org.apache....
1
vote
2
answers
369
views
Is the Spark Java framework compatible with Java 17?
Spark - a tiny web framework for Java 8
https://sparkjava.com/
It's always mentioned that this framework is for Java 8. Can anyone confirm that this framework is forward compatible? Can we use the ...
0
votes
1
answer
131
views
Rendering an image on a Java-Backend then displaying it in the Frontend
My goal is to have a node-editor in the frontend which passes information on how data is supposed to be rendered as an image to the backend, where the heavy lifting is done, resulting in an image ...
0
votes
1
answer
237
views
Convert a column of Spark dataframe to lowercase
How to convert a column value in Spark dataframe to lowercase/uppercase in Java?
For example, below is the input dataframe:
name | country | src | city | debit
----------------------------...
1
vote
2
answers
171
views
How can I filter directly in Cassandra before loading data into a Spark Dataset?
I want to filter data directly on Cassandra DB and load the filtered data using Spark Java. So far, I have only found .select() for selecting the rows and .filter() for filtering data after it has ...
0
votes
2
answers
62
views
Joining multiple spark dataframes on non-primary key
I am trying to join 1 parent Dataframe with 2 child Dataframes.
Here is how my parent DF look like
PersonId
FirstName
LastName
1
ABC
XYZ
Child DF 1
FirstName
FirstNameMatchedPersonIds
ABC
[1, 10, ...
0
votes
0
answers
65
views
spark-java certificate alias not selecting correct entry from keystore
I am trying to setup an embedded jetty webserver using sparkjava v2.9.4 and I am configuring the https secure protocol by calling spark.Service.secure(String keystore, String keystorePassword, String ...
1
vote
1
answer
125
views
spark java core - Is there a connection timeout default on server side for routes, if so can we override it?
I am trying to set the connection timeout for the spark server plain version as i want it to be inline with Client connection timeout, meaning if client times out the server request should not proceed ...
0
votes
2
answers
198
views
spark Connect Two Database tables to produce a third data
DataFrameLoadedFromLeftDatabase=data loaded using DataFrameReader from first database say LeftDB.
I need to
iterate through each row in this dataframe,
connect to a second database say RightDB,
find ...
1
vote
0
answers
30
views
Population of Dataset<Row> in java spark with reference from list of Maps
I have a Dataset<Row> df with columns (studentid,studentname,studentplace,student_strength)
Sample Output now :
studentid
studentname
studentplace
student_strength
1
Roberta
marketstreet
null
...
0
votes
0
answers
42
views
Why is request.contextPath null in Before filter?
I have the following Before filter:
before((req, res) -> log.info(req.requestMethod() + " " + req.contextPath()));
The documentation on http://sparkjava.com/documentation#filters it ...
0
votes
0
answers
60
views
Spark Java Returns 404 For All Endpoints
I have configured this endpoint and many others just like it. All endpoints return 404. I packaged my .jar in a kubernetes container. In the logs, I see "the requested route [/hello] has not ...
0
votes
1
answer
228
views
Spark reading json number as string
I'm working on spark java application and using spark 2.4.7 version. I have a json file that I'm loading into dataframe like
Dataset<Row> df = sparkSession().read().option("multiline",...
0
votes
1
answer
139
views
Error running spark simple app without installing spark
I am learning spark. And trying to run a simple spark app that output 5 dataframe rows without installing spark. I know that one can run spark app without installing spark. The app is throwing below ...
0
votes
1
answer
80
views
Sparkjava jetty server Android application problem access ssl secure webserver
I use this code keystor is .bks file (Android):
secure(KEYSTORE_FILE.getAbsolutePath(), KEYSTORE_PASSWORD,null,null);
get("/hello", (req, res) -> "Hello World");
When i try to ...
0
votes
1
answer
33
views
Need to return object before/ignoring path in spark java
I want to write my own very basic caching inside my sparkjava server.
My thought is to cache the response somewhere, and have some code in a before block that checks if the incoming path is one that ...
-1
votes
1
answer
325
views
Jetty 9 to 11 upgrade requires servlet 5.0. Latest spark-java still using servlet 3.0
Hi we're on our way to upgrade to jetty 11 and have come across this issue - Jetty 11 needs servlet 5.0 while spark-java even upto 2.9.4 is still using servlet 3.0. I could not find any info related ...
0
votes
2
answers
92
views
How to concatenate null columns in spark dataframe in java?
I am working with spark in java and I want to create a column which is a concatenation of all other column values separated by comma. I have tried few ways to do this but couldn't find a solution.
For ...
0
votes
0
answers
30
views
spark-submit How to pass argument to jar mentioned in --jar?
I have a PySpark project where to log microbatch metrics, we have a jar which is passed using --jars, it works but I need to add a on/off switch for logging. How can I pass this on off switch when ...
1
vote
0
answers
75
views
Spark Java - strange behavior when uploading files
In my project I want to try to upload files, here is the part of the code responsible for this:
MultipartConfigElement multipartConfigElement =
new MultipartConfigElement(
...
0
votes
1
answer
106
views
Apache Spark , detect key change on a row and group rows
I have the below dataframe in spark where I need to detect the key change ( on column rec) and create a new column called groupId. For example the first row and second row belong to one group until ...
1
vote
1
answer
191
views
Request body gets lost in http get request. (Java-Spark)
When I sent the following request to my program, the body gets lost:
GET http://localhost:4567/contacts/get-all
Authorization : test
Content-Type : text/plain
Hello
Java Code
public class Test {
...
0
votes
0
answers
77
views
sparkjava shutsdown randomly
We have an REST application developed in sparkjava.
Suddenly at some point it shutting down without any cause in the log.
July 1, 2022 17:43:27.087 [Thread-27] INFO spark.embeddedserver.jetty....
0
votes
1
answer
1k
views
How to create a struct column from a list of column names in Spark with Java?
I have a DataFrame with multiple columns, e.g.
root
|-- playerName
|-- country
|-- bowlingAvg
|-- bowlingSR
|-- wickets
|-- battingAvg
|-- battingSR
|-- runs
I also have a list of the column ...
-1
votes
1
answer
100
views
My SparkJava resource server gets 403 errors when trying to validate access tokens
I want to set up a very basic REST API using Spark-java, which just checks an access token obtained from my own authorisation server. It creates a GET request to the authorisation server's /oauth/...
4
votes
1
answer
997
views
How to union two dataframes which have same number of columns?
Dataframe df1 contains columns : a, b, c, d, e (Empty dataframe)
Dataframe df2 contains columns : b, c, d, e, _c4 (Contains Data)
I want to do a union on these two dataframes. I tried using
df1.union(...
1
vote
1
answer
117
views
How to create an edge list using a column of node path of ArrayType in Spark?
I have a Spark Dataset containing a single column of ArrayType which denotes the path from one user to another through their mutual friends
path
["Amy","John","Wally"]
...
0
votes
1
answer
96
views
Can't deploy SparkJava program on Heroku , getting Insufficient privileges error
I have a simple sparkJava endpoint, i am trying to deploy it on Heroku but i am getting the following error.
Failed to deploy application: Insufficient privileges to "spark-heroku-example" ...
0
votes
0
answers
124
views
Spark conversion error on the executor JDBC SQLSERVER partition option
I'm first reading lower bound and upper bound using a:
select max(timestamp) ,min(timestamp) from table name
extracting
Row row=query.collectasList().get(0).getString(0) as lowerbound and upperbound
...
0
votes
1
answer
614
views
Creating empty Spark dataframe and adding arbitrary values
I have created an empty dataframe and then trying to add columns and values to the dataframe.
Dataset<Row> runMetadata = sparkSession.emptyDataFrame();
runMetadata = runMetadata.withColumn("...
2
votes
1
answer
1k
views
FileNotFoundException (operation not permitted) error but signs point to file present?
I am trying to deploy a SparkJava REST app in a Kubernetes container on my Windows machine.
Windows 10
Kubernetes v1.22.5
(edit) base image: openjdk:8-alpine
I am trying to read in a properties file ...
2
votes
1
answer
173
views
Spark's DELETE handler returns NULL when accessing any request.queryParams key
I developed a very simple API using the Spark Framework (ref: sparkjava.com). I wrote a delete handler and tested it with Postman.
The Problem
When I try to assign a value using request.queryParams, ...
0
votes
0
answers
136
views
GET Method not fetching latest data i POSTED Java and Spark
So, this is the problem... when i insert new data to database using post method, it worked, the med table in database have new data. but when i fetch the data using GET Method, the data i inserted ...
0
votes
2
answers
686
views
Java Spark convert Empty values in Dataframe to null
I'm trying to convert all empty values in my spark Dataframe to null using:
df.withColumn(colname, when(df.col(colname).equalTo(""), null)
.otherwise(df....
1
vote
1
answer
738
views
IntelliJ cannot resolve symbol 'spark' when importing Spark Java
I am currently running the latest IntelliJ and I am trying to use SparkJava Webserver for a Gradle project I'm writing.
When I do compile and run with gradle everything works fine, however I am ...
0
votes
1
answer
382
views
Spark Dataframe MapType filter on Keys and Values
I am using spark 2.4 & Java.
I am playing around with MapTypes in spark.
Consider the following code
public class MapTypes {
public static void main(String[] args) {
SparkSession spark ...
0
votes
0
answers
128
views
Call service method from spark
I'm using spark with spring boot. I have a service ( spring bean ), which I want to call for batch of data. The issue is that I keep getting Task not serializable exception.
javaRDD.foreachPartition(...
0
votes
2
answers
1k
views
package spark does not exist
I'm just starting to learn Java.
I am using IntelliJ to create a SparkJava (see: sparkjava.com) app using openjdk 17. I followed the tutorial instructions outlined in "Setting up Spark with ...