Questions tagged [spark-submit]
spark-submit is a script that is able to run apache-spark code written in e.g. java, scala or python
598
questions
0
votes
0
answers
26
views
Spark local[*] with example
I am new to pyspark. I learnt that local[*] run Spark locally with as many worker threads as logical cores on your machine. can anyone explain this in detail, what it means?
For example, if I have a ...
-1
votes
1
answer
25
views
Spark doesn't see Hive tables depending on how you run it
The problem is that depending on how you run Spark you can see or not Hive databases. I do three next actions:
Using Hive
hive> show databases;
OK
default
mydb
sparkdb
Time taken: 0.041 seconds, ...
0
votes
0
answers
8
views
How spark submit with python in venv and local mode?
Hi i want run my python script with spark-submit with venv's python.
In my shell i digit:
export PYSPARK_PYTHON=/home/analytics/venv3.6/bin/python
spark-submit --master local /home/analytics/...
0
votes
0
answers
38
views
Spark job getting stuck forever
I am trying to run it in the docker. But it is getting stuck after giving output. Am i missing something?
FROM bitnami/spark:3
USER root
RUN curl https://repo1.maven.org/maven2/com/amazonaws/aws-...
0
votes
0
answers
32
views
hbase-spark connector for spark2.1.0
I am using below stack
Hadoop-2.7.7
spark-2.4.5
Hbase-2.1.0
zk-3.5.9
I want to read and write data on hbase using spark with spark-submit command. But i was unable to do so.
I have successfully ...
0
votes
1
answer
131
views
I am relatively new to Airflow and spark and want to use the Kubernetes Operator in airflow Dag to run a Spark Submit command
I am using the kubernetes version 1.25 client and server, I have deployed Airflow using the official helm charts on the environment. I want the Airflow dags kubernetes pod operator that has code to ...
0
votes
0
answers
61
views
Is it possible to run spark-submit task on databricks using archives parameter?
My problem is the following:
I'm trying to run a job with spark submit task, but I have an environment to build.
But for metro archives does not install my environment inside the cluster at runtime ...
0
votes
0
answers
36
views
Can we have multiple executors in Spark master local[*] deployment code client
I have a 1 node Hadoop Cluster, I am submitting a spark job like this
spark-submit \
--class com.compq.scriptRunning \
--master local[*] \
--deploy-mode client \
--num-executors 3 \
...
0
votes
0
answers
24
views
Memory,CPU,GPU profiling in containerized Spark cluster
any suggestion in which library/tool should I use for plotting over time RAM,CPU and (optionally) GPU usage of a spark-app submitted to a Docker containerized Spark cluster through spark-submit?
In ...
0
votes
1
answer
316
views
spark-sql/spark-submit with delta lake is resulting null pointer exception (at org.apache.spark.storage.BlockManagerMasterEndpoint)
I'm using delta lake on using pyspark by submitting below command
spark-sql --packages io.delta:delta-core_2.12:0.8.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --...
0
votes
0
answers
27
views
How to run a script residing in an image when running AWS EMR spark-submit from Lambda?
I have been provided a container image with a script and supporting files inside residing in /opt/spark/work-dir.
Example:
test-job.py
test-config.json
dataset/set1/* (This consist of more files and ...
0
votes
1
answer
73
views
Connection to remote Hadoop Cluster (CDP) through Linux server
I'm new to PySpark and I want to connect remote Hadoop Cluster (CDP) through Linux server by using spark-submit command.
Any help would be appreciated.
I need spark-submit command to connect remote ...
0
votes
0
answers
95
views
Using aws emr add-steps for spark-submit
We have a complicated spark-submit command that we would like to submit to AWS EMR using the aws emr add-steps CLI command. We are having trouble figuring out the correct syntax to use. For example, ...
1
vote
0
answers
34
views
Restrict executer and driver, memory and core parameter in spark-submit on spark only cluster
We have a spark only cluster for multiple users, in spark-default.conf have below property set
spark.driver.memory 2g
spark.executor.cores 1
spark.executor.memory 2g
As we have a multiple users, i ...
1
vote
1
answer
141
views
ModuleNotFoundError: No module named X when using foreach function with PySpark
I currently encounter an error when using an external Python module (orjson) inside foreach function with Pyspark. Everything was fine if I use that module outside foreach function (collect() method). ...
0
votes
0
answers
24
views
Spark Submit Does Not Fail If The SQL Added to Run It Has a Query That Fails
I have a spark submit command that is given a .sql file with 10 consecutive queries in it to run.
If one of the queries fails due to syntax error or table not found issue, the spark submit command ...
0
votes
0
answers
73
views
how to run a program in spark using spark-submit in cluster mode?
I am using spark-class org.apache.spark.deploy.master.Master to run master in cmd & also to get ip and port no. To start worker spark-class org.apache.spark.deploy.worker.Worker spark:////10.1.0.4:...
1
vote
1
answer
71
views
ClassNotFoundException: org.apache.beam.runners.spark.io.SourceRDD$SourcePartition during spark submit
I use spark-submit to spark standalone cluster to execute my shaded jar, however the executor gets error:
22/12/06 15:21:25 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1) (10.37.2.77, ...
0
votes
0
answers
22
views
How to use FS gateway handler for python
Currently I'm trying copy from local dir to HDFS directory using fs handler from java gateway in python, which will later the script will be running using spark-submit.
My First attempt:
fs....
0
votes
0
answers
28
views
how to generate jupyter notebook output in spark-submit?
I'm new in with Jupyter.
I have a python script that run on spark-submit, I need to show the output in html or pdf file. The output contains pandas tables and also should include some graphics.
I ...
1
vote
0
answers
51
views
spark-submit error - "Local jar does not exists, skipping"
Spark: 3.3.0 (spark-3.3.0-bin-hadoop3)
I am trying to run spark-submit using jar files, but I receive "jar does not exist" from DependencyUtils, even if the JAR is in place. As a consequence ...
0
votes
0
answers
123
views
Submit Spark jobs to local running Spark standalone cluster from Airflow running in Docker
I have Airflow 2.3.4 running in Docker containers and a Spark 3.3.0 standalone cluster running in my local machine. I am trying to use SparkSubmitOperator to submit a Spark job to my local Spark ...
1
vote
0
answers
107
views
Standard way to store/upload application jar on Spark cluster on Kubernetes
I have a Spark based Kubernetes cluster where I am using spark-submit to submit the jobs on cluster as needed.
e.g.
spark-submit \
--master spark://my-spark-master-svc:7077 \
--class com.Main \...
0
votes
0
answers
124
views
Virtualenv with relative path not found in Spark
I'm following this tutorial in order to manage my python dependencies in Spark. The idea is to bundle all my dependencies in a virtualenv and ship it with the --archives flag.
When I run this code (...
0
votes
0
answers
73
views
Making Spark and Slick work at the same time
I am writing a program in Scala and I came through this problem. I am trying to solve this for days now, and really getting on my nerve. But who knows maybe it is not even possible. I am doing a GUI ...
0
votes
0
answers
40
views
Accessing Folder Structure for Spark Submit zipped PyFile
I have a folder structure that looks like this:
├── repo
│ ├── src
│ └── main.py
│ ├── __init__.py
│ ├── utils
│ ├── __init__.py
│ └── helpers.py
│ └── ...
0
votes
0
answers
20
views
can driver program and cluster manager(resource manager) can available on same machine in spark stand alone
I'm doing spark submit from the same machine as spark master, using following command ./bin/spark-submit --master spark://ip:port --deploy-mode "client" test.py I'm my application running ...
0
votes
1
answer
205
views
spark-submit: NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException
the method belongs to SparkSession , the name is getOrCreate()
detailed exception is
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;...
0
votes
0
answers
14
views
Is there any way to submit job PySpark icluding extenal lib without pip install at servers?
I am currently working with spark by Scala, I easily package external lib in a jar file to submit a Job to YARN. I want to use pyspark but one thing that prevents me from using it is I have no ...
0
votes
1
answer
95
views
Stop spark log warning "Truncated the string representation of a plan ...."
I'm trying to use a log4j2 RegexFilter to filter the spark warning Truncated the string representation of a plan since it was too long. Spark logs this warning because I'm setting the config option ...
1
vote
1
answer
26
views
spark-submit error loading class with fatjar on macOS
I am trying to run a simple hello world spark application
This is my code
package com.sd.proj.executables
import org.apache.spark.sql.functions.lit
import org.apache.spark.sql.{DataFrame, ...
0
votes
2
answers
339
views
Spark configuration based on my data size
I know there's a way to configure a Spark Application based in your cluster resources ("Executor memory" and "number of Executor" and "executor cores") I'm wondering if ...
0
votes
0
answers
81
views
Spark Kafka Streaming not displaying data on spark-submit on EMR
I am trying to stream the data from a Kafka topic, it is working in spark shell. But if i create a .py file and use spark-submit for the same, it is failing:
Code:
spark_session = SparkSession.builder....
0
votes
0
answers
22
views
spark-submit from one EKS cluster to another EKS cluster
There are two AWS EKS clusters (EKS-1 and EKS-2).
Both are in the same VPC.
There is Spark cluster running on EKS-2.
On EKS-1, I need to run spark-submit with --master parameter, which points to EKS-2....
0
votes
1
answer
63
views
Spark cluster on Kubernetes without spark-submit
I have a spark application and want to deploy this on a Kubernetes cluster.
Following the below documentation I have managed to create an empty Kubernetes cluster, generated docker image using the ...
0
votes
1
answer
44
views
Spark Submit from SSH has different behavior
I have a yarn single node cluster setup in an Ubuntu VM.
When I am doing a spark-submit from the VM everything is working fine but when I am launching the same command from another VM with SSH the job ...
0
votes
1
answer
181
views
Error during spark submiting job on Yarn cluster from remote host
I try to spark-submit my jar with Spark application to remote Yarn Cluster.
I downloaded files from cluster:
hdfs-site.xml
yarn-site.xml
core-site.xml
Set environment variable HADOOP_CONF_DIR on ...
-1
votes
1
answer
81
views
Spark submit not working with protected nexus repository
Trying resolve builds from nexus not working if the nexus repo is password protected.
giving --repositories http://:<url_safe_password>@:/<repo_path> doesn't work
0
votes
1
answer
306
views
spark-submit to remote Spark
Currently I've deployed Spark to minikube in my local machine. Pod and its containers are up and running, and I've already checked that port 7077 is listening from the host machine (local machine).
...
0
votes
1
answer
247
views
how to submit a pyspark job in google cloud shell; how to pass files and arguments in pyspark submit; how to read them in pyspark code
Can anyone please help how to
submit a pyspark job in google cloud shell
pass files and arguments in pyspark submit
read those files and arguments in pyspark code
0
votes
1
answer
524
views
How to resolve: `pods is forbidden: User "system:anonymous" cannot watch resource "pods" in API group "" in the namespace "default"`
I want to use Kubernetes as resource manager for spark.
so I wanted to submit a jar far to spark cluster with spark-submit:
./bin/spark-submit \
--master k8s://https://vm:6443 \
--class com....
0
votes
0
answers
19
views
Pyspark script launch problems
I have set my MacOS Environment variables for PYSPARK_PYTHON and PYSPARK_DRIVER from my home-brew installation. Next, I want to run my main.py-file with 'spark-submit main.py'. However, the only ...
0
votes
1
answer
74
views
How to use checkpoints in azure Kubernetes for spark submit
How to configure spark job checkpoint in azure blob storage?
I was able to add checkpoints in Databricks but not in azure Kubernetes cluster
Can you please help me to overcome the issue
0
votes
1
answer
181
views
Spark-submit in dataproc cluster generating java.lang.ClassNotFoundException:
The spark-submit runs fine on a local cluster without any issues. Due to resource limitations, I switched to cloud-based computing. Currently, I am running a spark-cluster in Google Cloud Dataproc ...
0
votes
0
answers
143
views
Can we pass list of values for an argument in config file in pyspark?
I have config file that has all the configurations, shell script that has the spark-submit command, and the actual pyspark script
I want to apply a filter on a field to a df in python script.
Example
...
0
votes
1
answer
159
views
How to use --properties-file flag in dataproc?
While doing spark-submit, Gcloud gives an option to use --properties-file to pass the cluster properties and spark configurations. I am not sure how to use it while running the job.
1
vote
0
answers
268
views
spark-submit DependencyUtils: Local jar does not exists, skipping
Context: Windows 10/Linux Ubuntu 8 LTS
Spark: 3.3.0 (spark-3.3.0-bin-hadoop3)
I'm running spark-submit with a fat jar, but I receive a "jar does not exists" from DependencyUtils, even if the ...
0
votes
0
answers
49
views
spark-submit failed to load class on windows 10
Context:
Scala, Spark application, fat JAR, spark 3.3.0, windows 10
IDE:IntelliJ IDEA 2022.2.2
Package: it..<...>.MyPackage
MainClass : it..<...>.MyPackage.Application
Jar building: using ...
0
votes
1
answer
115
views
Spark with python script from console not working no errors
Hi I got this weird problem. I am new to spark, just installed spark and java, made the environmental variables and spark seems to work properly. I can enter the spark console form cmd and write print ...
-1
votes
2
answers
24
views
how to get spark-sbumit logging result
When i submit spark job too terminal, it's has logging result like image in terminal.
How can i get it and set it to value or object?