Questions tagged [spark-java]

Spark is a lightweight web application framework inspired by Ruby's Sinatra. DO NOT USE FOR QUESTIONS ABOUT APACHE SPARK.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Spark job throwing exceptions during loading data from Kafka to Hive

We have a big data cluster where we have data in Kakfa topics and we load it to hive using spark job (with java 8).I was using Cloudera 7.1.7 version and spark version (2.4.7.7.1.7.1000-141) SP1 and ...
Sumit Kumar's user avatar
-1 votes
1 answer
34 views

Spark Java sum is giving incorrect value

Spark Java sum is giving incorrect value Java sample code is as below List<Double> points = Arrays.asList(-6221.4, 6380.46); Dataset<Row> dt = spark.createDataset(points, Encoders.DOUBLE()...
Shyam's user avatar
  • 9
0 votes
0 answers
32 views

Unable to serve a static web page using Spark java framework

I have the below code - which is configured to serve static web page within the package (src/main/resources/public) Which is not working and i dont see any exception in the logs as well public void ...
rollno748's user avatar
  • 334
0 votes
0 answers
20 views

Redirects to another site

Can I make redirect from current page to google.com (for example)? I tried to use method redirect and header with Location. Is it possible to somehow implement a redirect to another website, or only ...
casetal's user avatar
  • 11
1 vote
1 answer
63 views

How to set schema into spark.sql.function.from_csv?

I use spark-3.4.1-hadoop3 on windows 11. And I try to generate the schema to pass into from_csv function parameter. Belows are my codes. import org.apache.spark.sql.Column; import org.apache.spark.sql....
Joseph Hwang's user avatar
  • 1,347
0 votes
1 answer
43 views

repartition not working with xml file in Spark

I have dataframe which I want to save as multiple xml files. This is my code: employees .repartition(col("first_name")) .write() .option(&...
Nemanja's user avatar
  • 3,295
0 votes
1 answer
61 views

How to get index/position of column in dataframe (Spark sql Java)

I am using Spark Java (not scala, python). I have to change my code so that my spark query will select all columns rather than a specific set of columns. (Like using select *). Before when I had a ...
chingu's user avatar
  • 117
0 votes
2 answers
42 views

Select a list of set columns AND other columns depending on their name/regex (Spark SQL Java)

I have a list of columns that I need to select. I have the field names for each column in this list, so it is easy to select. public Column[] getSelectColumns() { return new Column[]{ ...
chingu's user avatar
  • 117
0 votes
0 answers
66 views

Dynamically selecting column based on if that column exists in another column's value (Java Spark SQL)

I am using Spark SQL Java (Not scala or python). Here is a simplified example of my dataset (it has many more columns than this): ID References E_id F_id G_id 0 [E_id] 0000 1 [E_id, F_id, G_id] ...
chingu's user avatar
  • 117
0 votes
1 answer
52 views

How to get attribute value with correct datatype from Row in java spark sql

I am using Spark Java (not scala or python). I have a dataframe (Dataset<Row>) and I want to access specific fields from the Row. I have been able to get the field value if it is a String, but I ...
chingu's user avatar
  • 117
0 votes
0 answers
14 views

Is it possible to have executors evaluate part of the query instead of master finding the entire result set first in spark java?

I have 2 tables in postgres db that I would need to join and send the resulting output to kafka. To do that I wrote a java code that uses spark framework. The sample of my code is like so: Main class: ...
Joe's user avatar
  • 337
0 votes
2 answers
38 views

What is the correct way to use Regex_replace to replace a '|' with a '-' in Spark Java's column named details?

I have a column named details which has special character as | I want to replace it with - . How can I do it in spark Java? I have tried: regex_replace(details ,"|","-") ...
Sachin's user avatar
  • 1
0 votes
0 answers
59 views

Spark Error org.apache.spark.SparkException: Error communicating with MapOutputTracker during Spark submit

Getting below exception org.apache.spark.SparkException: Error communicating with MapOutputTracker at org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:523) at org.apache....
Apo's user avatar
  • 41
1 vote
2 answers
369 views

Is the Spark Java framework compatible with Java 17?

Spark - a tiny web framework for Java 8 https://sparkjava.com/ It's always mentioned that this framework is for Java 8. Can anyone confirm that this framework is forward compatible? Can we use the ...
Ajaya Krishna's user avatar
0 votes
1 answer
131 views

Rendering an image on a Java-Backend then displaying it in the Frontend

My goal is to have a node-editor in the frontend which passes information on how data is supposed to be rendered as an image to the backend, where the heavy lifting is done, resulting in an image ...
auf keinsten's user avatar
0 votes
1 answer
237 views

Convert a column of Spark dataframe to lowercase

How to convert a column value in Spark dataframe to lowercase/uppercase in Java? For example, below is the input dataframe: name | country | src | city | debit ----------------------------...
Monu's user avatar
  • 2,092
1 vote
2 answers
171 views

How can I filter directly in Cassandra before loading data into a Spark Dataset?

I want to filter data directly on Cassandra DB and load the filtered data using Spark Java. So far, I have only found .select() for selecting the rows and .filter() for filtering data after it has ...
AJay's user avatar
  • 11
0 votes
2 answers
62 views

Joining multiple spark dataframes on non-primary key

I am trying to join 1 parent Dataframe with 2 child Dataframes. Here is how my parent DF look like PersonId FirstName LastName 1 ABC XYZ Child DF 1 FirstName FirstNameMatchedPersonIds ABC [1, 10, ...
Azar's user avatar
  • 33
0 votes
0 answers
65 views

spark-java certificate alias not selecting correct entry from keystore

I am trying to setup an embedded jetty webserver using sparkjava v2.9.4 and I am configuring the https secure protocol by calling spark.Service.secure(String keystore, String keystorePassword, String ...
mpkusnierz's user avatar
1 vote
1 answer
125 views

spark java core - Is there a connection timeout default on server side for routes, if so can we override it?

I am trying to set the connection timeout for the spark server plain version as i want it to be inline with Client connection timeout, meaning if client times out the server request should not proceed ...
Wahid Abdullah's user avatar
0 votes
2 answers
198 views

spark Connect Two Database tables to produce a third data

DataFrameLoadedFromLeftDatabase=data loaded using DataFrameReader from first database say LeftDB. I need to iterate through each row in this dataframe, connect to a second database say RightDB, find ...
santhosh's user avatar
1 vote
0 answers
30 views

Population of Dataset<Row> in java spark with reference from list of Maps

I have a Dataset<Row> df with columns (studentid,studentname,studentplace,student_strength) Sample Output now : studentid studentname studentplace student_strength 1 Roberta marketstreet null ...
John Humanyun's user avatar
0 votes
0 answers
42 views

Why is request.contextPath null in Before filter?

I have the following Before filter: before((req, res) -> log.info(req.requestMethod() + " " + req.contextPath())); The documentation on http://sparkjava.com/documentation#filters it ...
RobbieS's user avatar
  • 120
0 votes
0 answers
60 views

Spark Java Returns 404 For All Endpoints

I have configured this endpoint and many others just like it. All endpoints return 404. I packaged my .jar in a kubernetes container. In the logs, I see "the requested route [/hello] has not ...
RobbieS's user avatar
  • 120
0 votes
1 answer
228 views

Spark reading json number as string

I'm working on spark java application and using spark 2.4.7 version. I have a json file that I'm loading into dataframe like Dataset<Row> df = sparkSession().read().option("multiline",...
Prateek Gautam's user avatar
0 votes
1 answer
139 views

Error running spark simple app without installing spark

I am learning spark. And trying to run a simple spark app that output 5 dataframe rows without installing spark. I know that one can run spark app without installing spark. The app is throwing below ...
perseverance's user avatar
0 votes
1 answer
80 views

Sparkjava jetty server Android application problem access ssl secure webserver

I use this code keystor is .bks file (Android): secure(KEYSTORE_FILE.getAbsolutePath(), KEYSTORE_PASSWORD,null,null); get("/hello", (req, res) -> "Hello World"); When i try to ...
user1400390's user avatar
0 votes
1 answer
33 views

Need to return object before/ignoring path in spark java

I want to write my own very basic caching inside my sparkjava server. My thought is to cache the response somewhere, and have some code in a before block that checks if the incoming path is one that ...
Russ Wheeler's user avatar
  • 2,590
-1 votes
1 answer
325 views

Jetty 9 to 11 upgrade requires servlet 5.0. Latest spark-java still using servlet 3.0

Hi we're on our way to upgrade to jetty 11 and have come across this issue - Jetty 11 needs servlet 5.0 while spark-java even upto 2.9.4 is still using servlet 3.0. I could not find any info related ...
shashwatZing's user avatar
  • 1,550
0 votes
2 answers
92 views

How to concatenate null columns in spark dataframe in java?

I am working with spark in java and I want to create a column which is a concatenation of all other column values separated by comma. I have tried few ways to do this but couldn't find a solution. For ...
Prateek Gautam's user avatar
0 votes
0 answers
30 views

spark-submit How to pass argument to jar mentioned in --jar?

I have a PySpark project where to log microbatch metrics, we have a jar which is passed using --jars, it works but I need to add a on/off switch for logging. How can I pass this on off switch when ...
user2552949's user avatar
1 vote
0 answers
75 views

Spark Java - strange behavior when uploading files

In my project I want to try to upload files, here is the part of the code responsible for this: MultipartConfigElement multipartConfigElement = new MultipartConfigElement( ...
Denis Fadeev's user avatar
0 votes
1 answer
106 views

Apache Spark , detect key change on a row and group rows

I have the below dataframe in spark where I need to detect the key change ( on column rec) and create a new column called groupId. For example the first row and second row belong to one group until ...
user avatar
1 vote
1 answer
191 views

Request body gets lost in http get request. (Java-Spark)

When I sent the following request to my program, the body gets lost: GET http://localhost:4567/contacts/get-all Authorization : test Content-Type : text/plain Hello Java Code public class Test { ...
xtay2's user avatar
  • 432
0 votes
0 answers
77 views

sparkjava shutsdown randomly

We have an REST application developed in sparkjava. Suddenly at some point it shutting down without any cause in the log. July 1, 2022 17:43:27.087 [Thread-27] INFO spark.embeddedserver.jetty....
Selvakumar Ponnusamy's user avatar
0 votes
1 answer
1k views

How to create a struct column from a list of column names in Spark with Java?

I have a DataFrame with multiple columns, e.g. root |-- playerName |-- country |-- bowlingAvg |-- bowlingSR |-- wickets |-- battingAvg |-- battingSR |-- runs I also have a list of the column ...
kvj's user avatar
  • 45
-1 votes
1 answer
100 views

My SparkJava resource server gets 403 errors when trying to validate access tokens

I want to set up a very basic REST API using Spark-java, which just checks an access token obtained from my own authorisation server. It creates a GET request to the authorisation server's /oauth/...
CiaranC94's user avatar
  • 176
4 votes
1 answer
997 views

How to union two dataframes which have same number of columns?

Dataframe df1 contains columns : a, b, c, d, e (Empty dataframe) Dataframe df2 contains columns : b, c, d, e, _c4 (Contains Data) I want to do a union on these two dataframes. I tried using df1.union(...
George's user avatar
  • 95
1 vote
1 answer
117 views

How to create an edge list using a column of node path of ArrayType in Spark?

I have a Spark Dataset containing a single column of ArrayType which denotes the path from one user to another through their mutual friends path ["Amy","John","Wally"] ...
Gorden Lim's user avatar
0 votes
1 answer
96 views

Can't deploy SparkJava program on Heroku , getting Insufficient privileges error

I have a simple sparkJava endpoint, i am trying to deploy it on Heroku but i am getting the following error. Failed to deploy application: Insufficient privileges to "spark-heroku-example" ...
Haseeb Ansari's user avatar
0 votes
0 answers
124 views

Spark conversion error on the executor JDBC SQLSERVER partition option

I'm first reading lower bound and upper bound using a: select max(timestamp) ,min(timestamp) from table name extracting Row row=query.collectasList().get(0).getString(0) as lowerbound and upperbound ...
user2515163's user avatar
0 votes
1 answer
614 views

Creating empty Spark dataframe and adding arbitrary values

I have created an empty dataframe and then trying to add columns and values to the dataframe. Dataset<Row> runMetadata = sparkSession.emptyDataFrame(); runMetadata = runMetadata.withColumn("...
mayank goyal's user avatar
2 votes
1 answer
1k views

FileNotFoundException (operation not permitted) error but signs point to file present?

I am trying to deploy a SparkJava REST app in a Kubernetes container on my Windows machine. Windows 10 Kubernetes v1.22.5 (edit) base image: openjdk:8-alpine I am trying to read in a properties file ...
RobbieS's user avatar
  • 120
2 votes
1 answer
173 views

Spark's DELETE handler returns NULL when accessing any request.queryParams key

I developed a very simple API using the Spark Framework (ref: sparkjava.com). I wrote a delete handler and tested it with Postman. The Problem When I try to assign a value using request.queryParams, ...
Robert S's user avatar
  • 496
0 votes
0 answers
136 views

GET Method not fetching latest data i POSTED Java and Spark

So, this is the problem... when i insert new data to database using post method, it worked, the med table in database have new data. but when i fetch the data using GET Method, the data i inserted ...
HamdanAs's user avatar
0 votes
2 answers
686 views

Java Spark convert Empty values in Dataframe to null

I'm trying to convert all empty values in my spark Dataframe to null using: df.withColumn(colname, when(df.col(colname).equalTo(""), null) .otherwise(df....
Aditya 's user avatar
1 vote
1 answer
738 views

IntelliJ cannot resolve symbol 'spark' when importing Spark Java

I am currently running the latest IntelliJ and I am trying to use SparkJava Webserver for a Gradle project I'm writing. When I do compile and run with gradle everything works fine, however I am ...
Druckermann's user avatar
0 votes
1 answer
382 views

Spark Dataframe MapType filter on Keys and Values

I am using spark 2.4 & Java. I am playing around with MapTypes in spark. Consider the following code public class MapTypes { public static void main(String[] args) { SparkSession spark ...
Ranganath Tirumala's user avatar
0 votes
0 answers
128 views

Call service method from spark

I'm using spark with spring boot. I have a service ( spring bean ), which I want to call for batch of data. The issue is that I keep getting Task not serializable exception. javaRDD.foreachPartition(...
Hakob Hakobyan's user avatar
0 votes
2 answers
1k views

package spark does not exist

I'm just starting to learn Java. I am using IntelliJ to create a SparkJava (see: sparkjava.com) app using openjdk 17. I followed the tutorial instructions outlined in "Setting up Spark with ...
RobbieS's user avatar
  • 120

1
2 3 4 5
12