Questions tagged [spark-jobserver]
spark-jobserver provides a RESTful interface for submitting and managing Apache Spark jobs, jars, and job contexts.
156
questions
17
votes
2answers
3k views
Comparing Apache Livy with spark-jobserver
I know Apache Livy is the rest interface for interacting with spark from anywhere. So what is the benefits of using Apache Livy instead of spark-jobserver. What are the drawbacks of spark-jobserver ...
12
votes
2answers
4k views
why Livy or spark-jobserver instead of a simple web framework?
I'm building a RESTful API on top of Apache Spark. Serving the following Python script with spark-submit seems to work fine:
import cherrypy
from pyspark.sql import SparkSession
spark = SparkSession....
6
votes
0answers
1k views
java.nio.channels.ClosedByInterruptException in Spark
I have a dockerized Spark instance and use SJS to submit spark jobs from my Spring boot application.
Everything worked fine so far but now I get the following exception when submitting a job as per ...
5
votes
1answer
2k views
Apache spark rest API
I'm using the spark-submit command I have for the log4j properties to invoke a Spark-submit like this:
/opt/spark-1.6.2-bin-hadoop2.6/bin/spark-submit \
--driver-java-options \
"-Dlog4j.configuration=...
5
votes
3answers
2k views
Deploy Apache Spark application from another application in Java, best practice
I am a new user of Spark. I have a web service that allows a user to request the server to perform a complex data analysis by reading from a database and pushing the results back to the database. I ...
4
votes
1answer
1k views
Spark Job Server Keep getting Invalid Jar
I did the build using sbt job-server-api/packages and sbt job-server-tests/packages.
I am able to extend/unzip the jar using jar -cxf command
curl --data-binary /opt/hadoop/spark-jobserver/job-server-...
4
votes
2answers
40k views
org.apache.spark.sql.AnalysisException: cannot resolve given input columns
exitTotalDF
.filter($"accid" === "dc215673-ef22-4d59-0998-455b82000015")
.groupBy("exiturl")
.agg(first("accid"), first("segment"), $"exiturl", sum("session"), sum("sessionfirst"), first("date"))...
4
votes
2answers
2k views
Pause Spark Streaming Job
I have simple spark streaming application which reads data from Kafka and then send this data after transformation on a http end point (or another kafka - for this question let's consider http). I am ...
4
votes
1answer
2k views
Connect Spark application with web server
I just will try to explain my simplified use case. There is:
Spark Application which count the words.
A web server which serves a web page with a form.
User that can type word within this form and ...
4
votes
2answers
574 views
Realtime request-based recommendations with Spark - Spark JobServer?
We are trying to find a way to load a Spark (2.x) ML trained model so that on request (through a REST interface) we can query it and get the predictions, e.g. http://predictor.com:8080/give/me/...
4
votes
1answer
782 views
Core Usage of Spark Streaming Applications
We create a system consisting of multiple Spark Streaming applications with each applications having multiple receivers. As far as I understood, each receivers needs its own core in the cluster. We ...
4
votes
0answers
355 views
ClassCastException when using Spark Dataset API+case class+Spark Job Server
I'm getting weird error whenever I re-create (delete and create context) the Spark SQL Context and run the job for 2nd time or after it will always throw this exception.
[2016-09-20 13:52:28,743] ...
4
votes
0answers
1k views
Spark job can not acquire resource from mesos cluster
I am using Spark Job Server (SJS) to create context and submit jobs.
My cluster includes 4 servers.
master1: 10.197.0.3
master2: 10.197.0.4
master3: 10.197.0.5
master4: 10.197.0.6
But only master1 ...
4
votes
1answer
758 views
The error “Invalid job type for this context” in spark SQL job with Spark job server
I create a spark SQL job with spark job server and use HiveContext following the sample below:
https://github.com/spark-jobserver/spark-jobserver/blob/master/job-server-extras/src/spark.jobserver/...
3
votes
3answers
3k views
Spark Job Server: “The server was not able to produce a timely response to your request”
I'm using Spark Job Server to run a Spark job and it works perfectly. But when I try to execute a big job (needs more than 40 sec) I get this error:
The server was not able to produce a timely ...
3
votes
2answers
2k views
Building spark-jobserver Using SBT and Scala
Can anyone suggest me a better documentation about spark-jobserver. I have gone through the url spark-jobserver but unable to follow the same. It will be great if some one explain step by step ...
3
votes
0answers
97 views
Spark Jobserver - stress test - Async POST error response: akka.pattern.AskTimeoutException
Hi I am trying to do stress test on the spark job server, and I am sharing the spark context with the following properties among the submitted jobs.
spark.executor.cores='2'
spark.cores.max='...
3
votes
0answers
555 views
spark jobserver job crash
I'm running a job using spark jobserver (takes +-10min).
The job randomly crash during its execution (around 1 time on 2) with the following exception on the executor :
ERROR 2016-10-13 19:22:58,617 ...
3
votes
2answers
1k views
Submitting Spark Jobs to Spark Cluster
I am a complete novice in Spark and just started exploring more on this. I have chosen the longer path by not installing hadoop using any CDH distribution and i have installed Hadoop from Apache ...
3
votes
0answers
507 views
How to process REPL generated class files in Spark by using parallel running Scala interpreters?
In my company we are currently using Spark interpreter to generate dynamically class files with spark-jobserver. Those class files are generated on our Spark cluster driver and saved into the ...
2
votes
2answers
2k views
Writing Parquet file in standalone mode works..multiple worker mode fails
In Spark, version 1.6.1 (code is in Scala 2.10), I am trying to write a data frame to a Parquet file:
import sc.implicits._
val triples = file.map(p => _parse(p, " ", true)).toDF()
triples.write....
2
votes
2answers
1k views
getting timeout when submitting fat jar to spark-jobserver (akka.pattern.AskTimeoutException)
I have built my job jar using sbt assembly to have all dependencies in one jar. When I try to submit my binary to spark-jobserver I am getting akka.pattern.AskTimeoutException
I modified my ...
2
votes
1answer
457 views
Spark JobServer, memory settings for release
I've set up a spark-jobserver to enable complex queries on a reduced dataset.
The jobserver executes two operations:
Sync with the main remote database, it makes a dump of some of the server's ...
2
votes
1answer
747 views
Spark Jobserver max-jobs-per-context
How would you determine a safe max threshold value for the max-jobs-per-context setting, which controls the number of concurrent Spark jobs that are running on a context? What would happen if you go ...
2
votes
3answers
809 views
spark jobserver ERROR classnotfoundexception
I have been trying spark using spark-shell. All my data is in sql.
I used to include external jars using the --jars flag like /bin/spark-shell --jars /path/to/mysql-connector-java-5.1.23-bin.jar --...
2
votes
1answer
1k views
How do I configure the FAIR scheduler with Spark-Jobserver?
When I post simultaneous jobserver requests, they always seem to be processed in FIFO mode. This is despite my best efforts to enable the FAIR scheduler. How can I ensure that my requests are always ...
2
votes
1answer
128 views
Deploying spark-jobserver to BlueMix Spark Node
I'm trying to deploy and connect instance of spark-jobserver in a docker container to BlueMix Spark service. Locally, container start perfectly with a command docker -d -p 8090:8090 {image-name}, but ...
2
votes
1answer
582 views
How to handle cassandra connections in spark job?
I am doing stress test on my spark application which uses spark cassandra connector as well as cassandra driver.
In my application , I am using cassandra driver to select the most recent value from ...
2
votes
1answer
312 views
spark jobserver failing to build with Spark 2.0
I am trying to run spark-jobserver with spark-2.0
I cloned branch spark-2.0-preview from github repository. I follow the deployment guide but when I try to deploy server using bin/server_deploy.sh. I ...
2
votes
0answers
120 views
Spark Jobserver: Very large task size
I'm getting messages along the lines of the following in my Spark JobServer logs:
Stage 14 contains a task of very large size (9523 KB). The maximum recommended task size is 100 KB.
I'm creating ...
2
votes
0answers
752 views
Apache Spark with Spark JobServer crash after some hours
I'm using Apache Spark 2.0.2 together with Apache JobServer 0.7.0.
I know this is not a best practice but this is a first step. My server have 52 Gb RAM and 6 CPU Cores, Cent OS 7 x64, Java(TM) SE ...
2
votes
0answers
563 views
Parallel execution of spark job using job-server
I'm using spark cluster in standalone mode + spark job-server for my written in Scala jobs execution. I launched job-server in docker container:
docker run -d -p 8090:8090 -e SPARK_MASTER=spark://...
2
votes
1answer
435 views
Schedule Automatic Spark Jobs in Spark Job server every hour
In DataStax Enterprise Edition 4.8 , Spark Jobserver 0.5.2 has been specially compiled against the supported version of Apache Spark 1.4.1.1. Spark job will read data from Cassandra and write ...
2
votes
0answers
194 views
Increase the Query Parallelism Capacity on Cached RDD (DataFrame) with Spark-Job-Server on a Standalone Spark Cluster
First of all, our standalone Spark cluster consists of 20 nodes, each one of them has 40 cores and 128G memory (including the 2 masters).
1.
We use Spark-Job-Server for reusing Spark-Context (in the ...
2
votes
0answers
469 views
How do I fix this Exception when starting spark-job-server?
I am trying to start the spark-job-server on my linux machine. I did the following:
Installed the cloudera distriution CDH(5.x) and gotten it up and running
Downloaded spark-job-server from the above ...
2
votes
2answers
548 views
Cassandra Connection Issue with Spark-jobserver
I am new to Spark world and Job Server
My Code :
package spark.jobserver
import java.nio.ByteBuffer
import scala.collection.JavaConversions._
import scala.collection.mutable.ListBuffer
import ...
1
vote
1answer
1k views
Persisting / Sharing a RDD in Spark Job Server
I want a RDD from a spark job to be persisted, so that it can be used by all subsequent jobs using Spark Job Server. Here is what i have tried:
Job 1:
package spark.jobserver
import com.typesafe....
1
vote
2answers
2k views
java.lang.UnsatisfiedLinkError: no mesos in java.library.path
I am trying to configure spark job-sever for Mesos cluster deployment mode.
I have set spark.master = "mesos://mesos-master:5050" in jobserver config.
When I am trying to create a context on job-...
1
vote
1answer
2k views
Run my SparkJob on apache job-server
I have a problem with Apache job-server and my .jar with SparkJob.
I have VirtualBox with DataStax. There are Cassandra and Spark. I install Apache job-server from git job-server. I want run examples ...
1
vote
1answer
693 views
How to run Spark Jobserver on Mesosphere's DC/OS
There is a lot I clearly don't understand about Spark, Spark Jobserver, and Mesosphere's DC/OS. But I very much like the Jobserver project, and also very much like our DC/OS cluster, and would really ...
1
vote
1answer
1k views
How to make a cached from a finished Spark Job still accessible for the other job?
My project is implement a interaction query for user to discover that data. Like we have a list of columns user can choose then user add to list and press view data. The current data store in ...
1
vote
1answer
359 views
Spark JobServer NullPointerException
I'm trying to start a spark jobserver, here are the steps I'm following:
I configure the local.sh based on the template.
Then I run ./bin/server_deploy.sh and it finishes without any error.
Configure ...
1
vote
1answer
427 views
Spark 2.1.1 with property spark.akka.threads = 12
I'm using Apache Spark 2.1.1 and Spark JobServer Spark 2.0 Preview.
I'm seeing on the spark UI Environment tab that there is a config property spark.akka.threads = 12 but in the documentation of ...
1
vote
2answers
977 views
In Spark JobServer. How to pass a json formated string on the input.string?
Im trying to execute the following curl command to run a job:
curl -k --basic --user 'user:psw' -d 'input.string= {"user":13}' 'https://localhost:8090/jobs?appName=test&classPath=test....
1
vote
1answer
8k views
'JavaPackage' object is not callable
My use case is as follows. I need to be able to call java methods from within python code
from py spark this seems to be very easy
I start the py spark like this
./pyspark --driver-class-...
1
vote
1answer
597 views
Spark flattening out dataframes
getting started with spark I would like to know how to flatmap or explode a dataframe.
It was created using df.groupBy("columName").count and has the following structure if I collect it:
[[Key1, ...
1
vote
1answer
517 views
Running a Java-based Spark Job on spark-jobserver
I need to run an aggregation Spark job using spark-jobserver using low-latency contexts. I have this Scala runner to run a job on using a Java method from a Java class.
object AggregationRunner ...
1
vote
1answer
472 views
Spark jobServer doesn't start
I've just got a weird problem when I attempt to instantiate the Spark Jobserver.
It seems it's looking at a wrong directory. Does anybody know something about this?. Maybe I am missing something but ...
1
vote
1answer
326 views
How to run sqlContext in the spark-jobserver
I'm trying to execute locally a job in the spark-jobserver. My application has the dependencies below:
name := "spark-test"
version := "1.0"
scalaVersion := "2.10.6"
resolvers += Resolver....
1
vote
1answer
187 views
Listener to get Job status from Spark Job Server
I am making rest requests to query Spark Job Server to get the status of the job. The code looks like below :
private Future<JobResult> getJobResultFuture(String jobId) {
...