Questions tagged [spark-submit]

spark-submit is a script that is able to run apache-spark code written in e.g. java, scala or python

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Spark local[*] with example

I am new to pyspark. I learnt that local[*] run Spark locally with as many worker threads as logical cores on your machine. can anyone explain this in detail, what it means? For example, if I have a ...
  • 157
-1 votes
1 answer
25 views

Spark doesn't see Hive tables depending on how you run it

The problem is that depending on how you run Spark you can see or not Hive databases. I do three next actions: Using Hive hive> show databases; OK default mydb sparkdb Time taken: 0.041 seconds, ...
0 votes
0 answers
8 views

How spark submit with python in venv and local mode?

Hi i want run my python script with spark-submit with venv's python. In my shell i digit: export PYSPARK_PYTHON=/home/analytics/venv3.6/bin/python spark-submit --master local /home/analytics/...
  • 1,270
0 votes
0 answers
38 views

Spark job getting stuck forever

I am trying to run it in the docker. But it is getting stuck after giving output. Am i missing something? FROM bitnami/spark:3 USER root RUN curl https://repo1.maven.org/maven2/com/amazonaws/aws-...
0 votes
0 answers
32 views

hbase-spark connector for spark2.1.0

I am using below stack Hadoop-2.7.7 spark-2.4.5 Hbase-2.1.0 zk-3.5.9 I want to read and write data on hbase using spark with spark-submit command. But i was unable to do so. I have successfully ...
0 votes
1 answer
131 views

I am relatively new to Airflow and spark and want to use the Kubernetes Operator in airflow Dag to run a Spark Submit command

I am using the kubernetes version 1.25 client and server, I have deployed Airflow using the official helm charts on the environment. I want the Airflow dags kubernetes pod operator that has code to ...
  • 31
0 votes
0 answers
61 views

Is it possible to run spark-submit task on databricks using archives parameter?

My problem is the following: I'm trying to run a job with spark submit task, but I have an environment to build. But for metro archives does not install my environment inside the cluster at runtime ...
0 votes
0 answers
36 views

Can we have multiple executors in Spark master local[*] deployment code client

I have a 1 node Hadoop Cluster, I am submitting a spark job like this spark-submit \ --class com.compq.scriptRunning \ --master local[*] \ --deploy-mode client \ --num-executors 3 \ ...
  • 59
0 votes
0 answers
24 views

Memory,CPU,GPU profiling in containerized Spark cluster

any suggestion in which library/tool should I use for plotting over time RAM,CPU and (optionally) GPU usage of a spark-app submitted to a Docker containerized Spark cluster through spark-submit? In ...
0 votes
1 answer
316 views

spark-sql/spark-submit with delta lake is resulting null pointer exception (at org.apache.spark.storage.BlockManagerMasterEndpoint)

I'm using delta lake on using pyspark by submitting below command spark-sql --packages io.delta:delta-core_2.12:0.8.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --...
  • 17
0 votes
0 answers
27 views

How to run a script residing in an image when running AWS EMR spark-submit from Lambda?

I have been provided a container image with a script and supporting files inside residing in /opt/spark/work-dir. Example: test-job.py test-config.json dataset/set1/* (This consist of more files and ...
0 votes
1 answer
73 views

Connection to remote Hadoop Cluster (CDP) through Linux server

I'm new to PySpark and I want to connect remote Hadoop Cluster (CDP) through Linux server by using spark-submit command. Any help would be appreciated. I need spark-submit command to connect remote ...
0 votes
0 answers
95 views

Using aws emr add-steps for spark-submit

We have a complicated spark-submit command that we would like to submit to AWS EMR using the aws emr add-steps CLI command. We are having trouble figuring out the correct syntax to use. For example, ...
  • 4,968
1 vote
0 answers
34 views

Restrict executer and driver, memory and core parameter in spark-submit on spark only cluster

We have a spark only cluster for multiple users, in spark-default.conf have below property set spark.driver.memory 2g spark.executor.cores 1 spark.executor.memory 2g As we have a multiple users, i ...
1 vote
1 answer
141 views

ModuleNotFoundError: No module named X when using foreach function with PySpark

I currently encounter an error when using an external Python module (orjson) inside foreach function with Pyspark. Everything was fine if I use that module outside foreach function (collect() method). ...
0 votes
0 answers
24 views

Spark Submit Does Not Fail If The SQL Added to Run It Has a Query That Fails

I have a spark submit command that is given a .sql file with 10 consecutive queries in it to run. If one of the queries fails due to syntax error or table not found issue, the spark submit command ...
0 votes
0 answers
73 views

how to run a program in spark using spark-submit in cluster mode?

I am using spark-class org.apache.spark.deploy.master.Master to run master in cmd & also to get ip and port no. To start worker spark-class org.apache.spark.deploy.worker.Worker spark:////10.1.0.4:...
  • 29
1 vote
1 answer
71 views

ClassNotFoundException: org.apache.beam.runners.spark.io.SourceRDD$SourcePartition during spark submit

I use spark-submit to spark standalone cluster to execute my shaded jar, however the executor gets error: 22/12/06 15:21:25 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1) (10.37.2.77, ...
0 votes
0 answers
22 views

How to use FS gateway handler for python

Currently I'm trying copy from local dir to HDFS directory using fs handler from java gateway in python, which will later the script will be running using spark-submit. My First attempt: fs....
  • 45
0 votes
0 answers
28 views

how to generate jupyter notebook output in spark-submit?

I'm new in with Jupyter. I have a python script that run on spark-submit, I need to show the output in html or pdf file. The output contains pandas tables and also should include some graphics. I ...
1 vote
0 answers
51 views

spark-submit error - "Local jar does not exists, skipping"

Spark: 3.3.0 (spark-3.3.0-bin-hadoop3) I am trying to run spark-submit using jar files, but I receive "jar does not exist" from DependencyUtils, even if the JAR is in place. As a consequence ...
0 votes
0 answers
123 views

Submit Spark jobs to local running Spark standalone cluster from Airflow running in Docker

I have Airflow 2.3.4 running in Docker containers and a Spark 3.3.0 standalone cluster running in my local machine. I am trying to use SparkSubmitOperator to submit a Spark job to my local Spark ...
1 vote
0 answers
107 views

Standard way to store/upload application jar on Spark cluster on Kubernetes

I have a Spark based Kubernetes cluster where I am using spark-submit to submit the jobs on cluster as needed. e.g. spark-submit \ --master spark://my-spark-master-svc:7077 \ --class com.Main \...
  • 378
0 votes
0 answers
124 views

Virtualenv with relative path not found in Spark

I'm following this tutorial in order to manage my python dependencies in Spark. The idea is to bundle all my dependencies in a virtualenv and ship it with the --archives flag. When I run this code (...
  • 232
0 votes
0 answers
73 views

Making Spark and Slick work at the same time

I am writing a program in Scala and I came through this problem. I am trying to solve this for days now, and really getting on my nerve. But who knows maybe it is not even possible. I am doing a GUI ...
  • 1
0 votes
0 answers
40 views

Accessing Folder Structure for Spark Submit zipped PyFile

I have a folder structure that looks like this: ├── repo │ ├── src │ └── main.py │ ├── __init__.py │ ├── utils │ ├── __init__.py │ └── helpers.py │ └── ...
0 votes
0 answers
20 views

can driver program and cluster manager(resource manager) can available on same machine in spark stand alone

I'm doing spark submit from the same machine as spark master, using following command ./bin/spark-submit --master spark://ip:port --deploy-mode "client" test.py I'm my application running ...
0 votes
1 answer
205 views

spark-submit: NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException

the method belongs to SparkSession , the name is getOrCreate() detailed exception is java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;...
0 votes
0 answers
14 views

Is there any way to submit job PySpark icluding extenal lib without pip install at servers?

I am currently working with spark by Scala, I easily package external lib in a jar file to submit a Job to YARN. I want to use pyspark but one thing that prevents me from using it is I have no ...
0 votes
1 answer
95 views

Stop spark log warning "Truncated the string representation of a plan ...."

I'm trying to use a log4j2 RegexFilter to filter the spark warning Truncated the string representation of a plan since it was too long. Spark logs this warning because I'm setting the config option ...
1 vote
1 answer
26 views

spark-submit error loading class with fatjar on macOS

I am trying to run a simple hello world spark application This is my code package com.sd.proj.executables import org.apache.spark.sql.functions.lit import org.apache.spark.sql.{DataFrame, ...
  • 31
0 votes
2 answers
339 views

Spark configuration based on my data size

I know there's a way to configure a Spark Application based in your cluster resources ("Executor memory" and "number of Executor" and "executor cores") I'm wondering if ...
0 votes
0 answers
81 views

Spark Kafka Streaming not displaying data on spark-submit on EMR

I am trying to stream the data from a Kafka topic, it is working in spark shell. But if i create a .py file and use spark-submit for the same, it is failing: Code: spark_session = SparkSession.builder....
0 votes
0 answers
22 views

spark-submit from one EKS cluster to another EKS cluster

There are two AWS EKS clusters (EKS-1 and EKS-2). Both are in the same VPC. There is Spark cluster running on EKS-2. On EKS-1, I need to run spark-submit with --master parameter, which points to EKS-2....
  • 3,197
0 votes
1 answer
63 views

Spark cluster on Kubernetes without spark-submit

I have a spark application and want to deploy this on a Kubernetes cluster. Following the below documentation I have managed to create an empty Kubernetes cluster, generated docker image using the ...
  • 378
0 votes
1 answer
44 views

Spark Submit from SSH has different behavior

I have a yarn single node cluster setup in an Ubuntu VM. When I am doing a spark-submit from the VM everything is working fine but when I am launching the same command from another VM with SSH the job ...
  • 13
0 votes
1 answer
181 views

Error during spark submiting job on Yarn cluster from remote host

I try to spark-submit my jar with Spark application to remote Yarn Cluster. I downloaded files from cluster: hdfs-site.xml yarn-site.xml core-site.xml Set environment variable HADOOP_CONF_DIR on ...
-1 votes
1 answer
81 views

Spark submit not working with protected nexus repository

Trying resolve builds from nexus not working if the nexus repo is password protected. giving --repositories http://:<url_safe_password>@:/<repo_path> doesn't work
0 votes
1 answer
306 views

spark-submit to remote Spark

Currently I've deployed Spark to minikube in my local machine. Pod and its containers are up and running, and I've already checked that port 7077 is listening from the host machine (local machine). ...
0 votes
1 answer
247 views

how to submit a pyspark job in google cloud shell; how to pass files and arguments in pyspark submit; how to read them in pyspark code

Can anyone please help how to submit a pyspark job in google cloud shell pass files and arguments in pyspark submit read those files and arguments in pyspark code
0 votes
1 answer
524 views

How to resolve: `pods is forbidden: User "system:anonymous" cannot watch resource "pods" in API group "" in the namespace "default"`

I want to use Kubernetes as resource manager for spark. so I wanted to submit a jar far to spark cluster with spark-submit: ./bin/spark-submit \ --master k8s://https://vm:6443 \ --class com....
0 votes
0 answers
19 views

Pyspark script launch problems

I have set my MacOS Environment variables for PYSPARK_PYTHON and PYSPARK_DRIVER from my home-brew installation. Next, I want to run my main.py-file with 'spark-submit main.py'. However, the only ...
0 votes
1 answer
74 views

How to use checkpoints in azure Kubernetes for spark submit

How to configure spark job checkpoint in azure blob storage? I was able to add checkpoints in Databricks but not in azure Kubernetes cluster Can you please help me to overcome the issue
0 votes
1 answer
181 views

Spark-submit in dataproc cluster generating java.lang.ClassNotFoundException:

The spark-submit runs fine on a local cluster without any issues. Due to resource limitations, I switched to cloud-based computing. Currently, I am running a spark-cluster in Google Cloud Dataproc ...
0 votes
0 answers
143 views

Can we pass list of values for an argument in config file in pyspark?

I have config file that has all the configurations, shell script that has the spark-submit command, and the actual pyspark script I want to apply a filter on a field to a df in python script. Example ...
  • 1
0 votes
1 answer
159 views

How to use --properties-file flag in dataproc?

While doing spark-submit, Gcloud gives an option to use --properties-file to pass the cluster properties and spark configurations. I am not sure how to use it while running the job.
1 vote
0 answers
268 views

spark-submit DependencyUtils: Local jar does not exists, skipping

Context: Windows 10/Linux Ubuntu 8 LTS Spark: 3.3.0 (spark-3.3.0-bin-hadoop3) I'm running spark-submit with a fat jar, but I receive a "jar does not exists" from DependencyUtils, even if the ...
  • 256
0 votes
0 answers
49 views

spark-submit failed to load class on windows 10

Context: Scala, Spark application, fat JAR, spark 3.3.0, windows 10 IDE:IntelliJ IDEA 2022.2.2 Package: it..<...>.MyPackage MainClass : it..<...>.MyPackage.Application Jar building: using ...
  • 256
0 votes
1 answer
115 views

Spark with python script from console not working no errors

Hi I got this weird problem. I am new to spark, just installed spark and java, made the environmental variables and spark seems to work properly. I can enter the spark console form cmd and write print ...
-1 votes
2 answers
24 views

how to get spark-sbumit logging result

When i submit spark job too terminal, it's has logging result like image in terminal. How can i get it and set it to value or object?

1
2 3 4 5
12