Questions tagged [spark-jobserver]
spark-jobserver provides a RESTful interface for submitting and managing Apache Spark jobs, jars, and job contexts.
156
questions
0
votes
0answers
42 views
How does spark-jobserver communicate with yarn?
I'm pretty new to spark and yarn and scala, and I'm looking at spark-jobserver to try to understand how the remote job submitting and monitoring works. What puzzles me is how spark-jobserver receives ...
1
vote
0answers
78 views
Job submitted via Spark job server fails with error
I am using Spark Job Server to submit spark jobs in cluster .The application I am trying to test is a
spark program based on Sansa query and Sansa stack . Sansa is used for scalable processing of ...
0
votes
0answers
23 views
How to work with spark-jobserver in azure databricks?
I'm looking for a way to work with databricks spark clusters interactively from a web application. Is there a way to configure spark-jobserver on azure databricks so that when a user fills a form in ...
0
votes
1answer
49 views
Spark Performance Monitoring
I have got a requirement to show the management/ Client that the executor-memory, number of cores, default parallelism, number of shuffle partitions and other configuration properties for running the ...
1
vote
0answers
30 views
Spark in-memory cache for metadata & block locations
In sake of Spark low latency jobs, Spark Job Server provides a Persistent Context option. But I'm not sure, does persistent context contains metadata, block locations & any other information ...
0
votes
1answer
40 views
Spark JobServer can use Cassandra as SharedDb
I have been doing a research about Configuring Spark JobServer Backend (SharedDb) with Cassandra.
And I saw in the SJS documentation that they cited Cassandra as one of the Shared DBs that can be used....
0
votes
1answer
91 views
How to import spark.jobserver.SparkSessionJob
I have added job-server 0.9.0 dependencies in build.sbt by add
scalaVersion := "2.11.0"
resolvers += "Job Server Bintray" at "https://dl.bintray.com/spark-jobserver/maven"
libraryDependencies ++= ...
0
votes
1answer
37 views
CONTEXT_ID added in SJS 0.9.0 is being set as null in the table
Im trying to catch up with the new SJS 0.9.0 in my application. Once after the context is created , I am trying to submit a job -> this happens
19/04/10 21:45:06 ERROR JobDAOActor: About to restart ...
0
votes
1answer
101 views
why spark always try to connect to localhost:9042?
When I build context I used following parameters:
spark.cassandra.connection.host=somehosts&spark.cassandra.auth.username=app&spark.cassandra.auth.password=app
more detail as follow:
{{tstr}...
0
votes
0answers
266 views
Node JS integration with spark job server
I'm trying to call spark job server API from node js. The API which is the python egg file does provide the count of nulls from the file. So once I call the API from the node, it is reaching the SJS ...
0
votes
0answers
2k views
Failed to create database “metastore_db” using spark-jobserver 0.8.1
I am upgrading my server to spark 2.3.0 and job-server 0.8.1-SNAPSHOT from spark 2.1.1 and job-server 0.8.0 (which were working fine). I am using the JobSqlDao with MySql and am using the ...
17
votes
2answers
3k views
Comparing Apache Livy with spark-jobserver
I know Apache Livy is the rest interface for interacting with spark from anywhere. So what is the benefits of using Apache Livy instead of spark-jobserver. What are the drawbacks of spark-jobserver ...
1
vote
0answers
49 views
sparkJobServer - running spark sql
It looks like spark.jobserver.context.SQLContextFactory is deprecated. Could somebody help me with the example on how to run Spark SQL with the latest ( 0.8) version of SparkJobServer.
Thank you.
1
vote
0answers
82 views
spark-jobServer 0.8.x SQL examples
I have installed spark-observer v 0.8.0 with Scala 2.11. I am able to run examples from job-server-tests. However Im unable to find any examples related to running SQL ( load some file/create ...
0
votes
1answer
142 views
What setup is needed to use the Spark Cassandra Connector with Spark Job Server
I am working with Spark and Cassandra and in general things are straight forward and working as intended; in particular the spark-shell and running .scala processes to get results.
I'm now looking at ...
0
votes
1answer
117 views
spark job server api dependency not found. what can i do?
Im putting the following in my pom.xml file:
<dependency>
<groupId>spark.jobserver</groupId>
<artifactId>job-server-api_2.11</artifactId>
<version>0.8....
1
vote
0answers
244 views
Can we have spark-jobserver in windows?
Is it possible to run spark-jobserver in windows without using any emulator like cygwin ? I have tried gitbash as well, as I thought it supports .sh files but I didn't get any luck.
Note : I have ...
0
votes
1answer
337 views
How to get percentage of job completion for a Spark Job?
I have been looking for a way to get percentage of Job completed for the corresponding job id.
Right now, the Spark JobServer UI shows the corresponding status for a running job:
{
"duration": "...
3
votes
0answers
97 views
Spark Jobserver - stress test - Async POST error response: akka.pattern.AskTimeoutException
Hi I am trying to do stress test on the spark job server, and I am sharing the spark context with the following properties among the submitted jobs.
spark.executor.cores='2'
spark.cores.max='...
0
votes
1answer
335 views
Why there is no support for sparkSession with namedObject in spark job server?
I am trying to build an application with spark job server API(for spark 2.2.0). But I found that there is no support for namedObject with sparkSession.
my looks like:
import com.typesafe.config....
0
votes
1answer
550 views
reusable sparkcontext instance
I'm quite new to Big Data and currently, I'm working on a CLI project that performs some text parsing using apache spark.
When a command is typed, a new sparkcontext is instantiated and some files ...
0
votes
1answer
267 views
spark job server does not return a json in the correct format
case class Response(jobCompleted:String,detailedMessage:String)
override def runJob(sc: HiveContext, runtime: JobEnvironment, data:
JobData): JobOutput = {
val generateResponse= new ...
0
votes
1answer
297 views
Spark Jobserver High Available
I have an standalone Spark cluster with few nodes. I was able to get it High Available with zookeeper. Im using Spark Jobserver spark-2.0-preview and I have configured the jobserver env1.conf file ...
0
votes
1answer
114 views
Cast from case class Named RDD
I was sharing my RDDs between jobs with type of CassandraRow but I'm now joining several RDDs together so a case class makes more sense.
I save my RDD as below & then retrieve it in a new job. ...
0
votes
1answer
117 views
spark-jobserver: Worker does not connect back to the driver
I set up a small Spark environment on two machines. One runs a master and a worker, and the other one runs a worker only. I can use this cluster using the Spark Shell like:
spark-shell --master spark:...
0
votes
1answer
129 views
Unable to start the deployed spark job server error org.slf4j.LoggerFactory not found
I am trying to use the spark job server on a cluster on CDH 5.11, spark version 1.6.0
When I try to start the spark jobserver on the deployed machine I get this error log
[ERROR] [06/02/2017 15:30:14....
1
vote
0answers
370 views
Getting java.lang.OutOfMemoryError: Java heap space in Spark job server
Getting java.lang.OutOfMemoryError: Java heap space in Spark job server logs and the job server goes down :
[2017-06-01 19:09:26,708] ERROR akka.actor.ActorSystemImpl [] [ActorSystem(JobServer)] - ...
0
votes
0answers
299 views
Spark Jobserver fail just by receiving a job request
Jobserver 0.7.0 it have 4Gb ram available and 10Gb for the context, the system have 3 more free Gb. The context was running for a while and at the time when receive a request fails without any error. ...
0
votes
1answer
266 views
Spark Job Server multithreading and dynamic allocation
I had pretty big expectations from Spark Job Server, but found out it critically lack of documentation.
Could you please answer one/all of next questions:
Does Spark Job Server submit jobs through ...
1
vote
1answer
427 views
Spark 2.1.1 with property spark.akka.threads = 12
I'm using Apache Spark 2.1.1 and Spark JobServer Spark 2.0 Preview.
I'm seeing on the spark UI Environment tab that there is a config property spark.akka.threads = 12 but in the documentation of ...
0
votes
1answer
79 views
Split RDD into many RDDs and Cache
I have an rdd like so
(aid, session, sessionnew, date)
(55-BHA, 58, 15, 2017-05-09)
(07-YET, 18, 5, 2017-05-09)
(32-KXD, 27, 20, 2017-05-09)
(19-OJD, 10, 1, 2017-05-09)
(55-BHA, 1, 0, 2017-05-09)
(...
4
votes
2answers
40k views
org.apache.spark.sql.AnalysisException: cannot resolve given input columns
exitTotalDF
.filter($"accid" === "dc215673-ef22-4d59-0998-455b82000015")
.groupBy("exiturl")
.agg(first("accid"), first("segment"), $"exiturl", sum("session"), sum("sessionfirst"), first("date"))...
1
vote
1answer
279 views
Error while compiling spark-jobserver
While refreshing my SBT project in IntelliJ IDEA, I am facing the following error:
Error:Error while importing SBT project:<br/>...<br/><pre>[warn] Credentials file C:\Users\usr\....
0
votes
0answers
194 views
ERROR .jobserver.JobManagerActor [] [] - About to restart actor due to exception: java.lang.NullPointerException
I got this error when I fallow this link: https://support.instaclustr.com/hc/en-us/articles/214473617-Getting-started-with-Spark-Jobserver-and-Instaclustr.
I am trying to write a service that I can ...
0
votes
0answers
287 views
Spark Jobserver too slow respect Apache Spark Queries
Using Jobserver Spark-2.0-preview and Apache Spark 2.1.1
The execution time per query is not greater than 1s when I check on the spark UI, but I receive the response from jobserver after 10 seconds ...
1
vote
0answers
122 views
Parallelism in Spark Job server
We are working on Qubole with Spark version 2.0.2.
We have a multi-step process in which all the intermediate steps write their output to HDFS and later this output is used in the reporting layer.
...
0
votes
0answers
189 views
How to keep log files for each job with created spark jobserver context
I used to run spark job server from server_start.sh, It comes with log files with default assigned context specified in log4j.
However, when I ran the following command created context
curl -d "" '...
1
vote
1answer
123 views
Spark and Spark JobServer best practice to avoid of expensive reconstruction objects in each call of runJob(sc: SparkContext, config: Config)
Every-time invoke JobServer runJob API below, I have a time-cost logic to construct an object repeatedly inside the runJob call
runJob(sc: SparkContext, config: Config)
What is the best practice to ...
1
vote
2answers
2k views
java.lang.UnsatisfiedLinkError: no mesos in java.library.path
I am trying to configure spark job-sever for Mesos cluster deployment mode.
I have set spark.master = "mesos://mesos-master:5050" in jobserver config.
When I am trying to create a context on job-...
1
vote
0answers
190 views
classPath spark.jobserver. not found
I have added a new code file along with the existing example files in the spark-jobserver directory and created a .jar. It throws the following error even after the jar file is uploaded successfully.
...
2
votes
2answers
1k views
getting timeout when submitting fat jar to spark-jobserver (akka.pattern.AskTimeoutException)
I have built my job jar using sbt assembly to have all dependencies in one jar. When I try to submit my binary to spark-jobserver I am getting akka.pattern.AskTimeoutException
I modified my ...
0
votes
1answer
174 views
Did start spark job server properly and out of memory response
I am using spark-jobserver-0.6.2-spark-1.6.1
(1) export OBSERVER_CONFIG = /custom-spark-jobserver-config.yml
(2)./server_start.sh
Execution of the above start shell file returns without error. ...
0
votes
0answers
71 views
Implementing Checkpointing in Spark Streaming Job submitted using Spark Job Server
Implementing checkpointing when spark streaming job is diretcly submitted to spark seems straight forward . We are a facing quite some complexities when we need to the same when the streaming job is ...
0
votes
1answer
91 views
Job submission fails with generic 503 error when max-jobs-per-context value exceeds
I have set context-per-jvm = true and max-jobs-per-context = 4 in my config file.
I have pre-created context using following curl command:
curl -d "" 'http://jobserver:8090/contexts/test-context'
...
1
vote
0answers
314 views
Spark job server starts but shows error (in akka.util.Helpers)
I have deployed spark job server (version 0.6.2) on a remote machine running a YARN cluster using Cloudera (version 5.8.2). Followed the instructions given here. After deploying, when I tried to start ...
2
votes
1answer
312 views
spark jobserver failing to build with Spark 2.0
I am trying to run spark-jobserver with spark-2.0
I cloned branch spark-2.0-preview from github repository. I follow the deployment guide but when I try to deploy server using bin/server_deploy.sh. I ...
2
votes
0answers
120 views
Spark Jobserver: Very large task size
I'm getting messages along the lines of the following in my Spark JobServer logs:
Stage 14 contains a task of very large size (9523 KB). The maximum recommended task size is 100 KB.
I'm creating ...
0
votes
0answers
269 views
Spark Jobserver Executor application finished with state KILLED exitStatus 0
Using Apache Spark 2.0.1 deployed as Standalone together with jobserver 0.7.0.
I have a small job to test if the context is operational due sometimes the contexts is killed but the java process on my ...
1
vote
1answer
326 views
How to run sqlContext in the spark-jobserver
I'm trying to execute locally a job in the spark-jobserver. My application has the dependencies below:
name := "spark-test"
version := "1.0"
scalaVersion := "2.10.6"
resolvers += Resolver....
4
votes
2answers
575 views
Realtime request-based recommendations with Spark - Spark JobServer?
We are trying to find a way to load a Spark (2.x) ML trained model so that on request (through a REST interface) we can query it and get the predictions, e.g. http://predictor.com:8080/give/me/...