Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
37 views

How can I read PySpark code from S3 buckets?

I have my Apache Livy setup and PySpark setup in AWS and I have kept my PySpark code in S3 bucket. Here is my code to submit via Livy: import requests import json livy_url_battches = "http://56....
arbind Sah's user avatar
0 votes
0 answers
15 views

How to check spark job on Amazon EMR console by job submit with lvy

I can successfully submit spark job through Apache livy to Amazon EMR. But I did not see any steps on Amazon EMR console. Though my spark job was completed and success, I could only check job state ...
Weever's user avatar
  • 25
0 votes
0 answers
30 views

AWS EMR Jupyterhub notebook run fails with error: Session isn't active

I noticed this question is asked a lot but the solution proposed in other threads do not work for me. I have created an EMR cluster on AWS and I am running a quite time intensive code. The notebook ...
Francesco Pegoraro's user avatar
0 votes
1 answer
40 views

Can Spark SQL utilize previous results

I am curious if Spark can utilize previous results to help execution, for the example below: Inside same session, I first create a view1 with some contents. Because there is no action, nothing is ...
user2988877's user avatar
0 votes
0 answers
80 views

How to use livy with external library

I try to use livy statement to submit a piece of scala code that reads lucene index and then read S3 files. As you can imagine, I try to use spark/livy to search data. Is that possible? How to include ...
user2988877's user avatar
0 votes
1 answer
49 views

Why Livy does not accept a large JSON args

Context: I have a C# API that sends an HTTP POST request to the /batches route of Livy, and Livy forwards the arguments to my Scala Spark driver. As far as I know, internally Livy runs the spark-...
Giovanna Bueno's user avatar
0 votes
0 answers
49 views

facing error while creating spark session using apache livy ERROR RestSubmissionClient: Exception from the cluster

ERROR RestSubmissionClient: Exception from the cluster: java.nio.file.NoSuchFileException: spark-internal Facing issue while creating spark session using Apache Livy. Below are versions using. apache-...
Swapnil Shinde's user avatar
0 votes
0 answers
20 views

during high load on yarn cluster, spark session take more time to initialize

I observed that spark session take time to initialize when there is high load on cluster , can someone help me to understand why this happen, is spark maintain session pool somewhere and during high ...
Shalaj's user avatar
  • 591
0 votes
0 answers
34 views

Livy session to submit pyspark from HDFS

I am trying to run a simple spark sql basically to extract data from a hive table using Livy 0.8.0 using below code. Firewall is already enabled for Livy url. Import requests from requests_kerberos ...
user1768029's user avatar
0 votes
0 answers
83 views

Can not send livy rest api using proxy user into kerberized hadoop cluster

We want to use proxyuser in livy rest api for processing the data in kerberized hadoop cluster. i.e. we want to impersonate user roslinda.statistik with livy user as below curl -X POST -H "...
m hanif f's user avatar
  • 418
2 votes
1 answer
99 views

Apache Livy session showing Application id NULL

Whenever I try to fetch the status of the livy application which ran on a GCP Dataproc cluster, It always returned me null value, irrespective what I changed in the livy.conf, This is bugging me, I ...
ash_ketchum12's user avatar
0 votes
0 answers
46 views

Accessing Dataproc Cluster through Apache Livy?

Can i access Dataproc cluster from Apache Livy instead of Dataproc API? If i can, Where I have to install my apache livy server? Currently we need to access Dataproc cluster through apache livy ...
Samy's user avatar
  • 1
0 votes
0 answers
58 views

Python and Livy compatible version

I have a VM in which livy server is running, and livy version is 0.7.1 and spark version is 2.4.8 and I'm using python 3.7 version Can I upgrade the my python version to >=3.8, will that be ...
Chithal's user avatar
  • 57
0 votes
0 answers
223 views

Why Livy session is not found after YARN application succeed?

Using pylivy I create session and download resulting dataframe. The running code itself is not important: with LivySession.create(self.LIVY_URL, kind=SessionKind.PYSPARK, requests_session=self....
Makrushin Evgenii's user avatar
-1 votes
1 answer
259 views

TypeError: required field "type_ignores" missing from Module Error Using Spark, Livy, Sparkmagic

where if I run my code as follows in separate cells line-by-line, there is no error like using Jupyter notebook code block import sys print(sys.version) 3.8.9 .. But, when i write code together ...
yeov's user avatar
  • 1
0 votes
0 answers
107 views

How to determine the practical character limit of a Livy request to submit Spark batch job sent from Airflow?

I am trying to figure out the upper limit (practical estimate is fine) of characters/bytes passed on as args to the LivyOperator in Airflow. The operator eventually uses args(: Sequence[str | int | ...
gCW's user avatar
  • 1
0 votes
1 answer
175 views

org.apache.hadoop.yarn.api.ApplicationClientProtocolPB is unauthorized Error?

I have kerberized Hadoop I set this in core-site.xml <property> <name>hadoop.proxyuser.hue.hosts</name> <value>*</value> </property> <property> ...
CompEng's user avatar
  • 7,326
0 votes
0 answers
872 views

EOL/Obsolete Software: Apache Log4j 1.X Detected

currently we are having Apache Log4j 1.X on one of our docker image which is reported as EOL and needs to update to Apache Log4j 2.X version but we are unable to find that package to upgrade it to ...
Shraddha Nemane's user avatar
0 votes
1 answer
427 views

How to pass Spark executor parameters to DBT?

I am using DBT with spark-livy-adapter. And I want to pass spark-conf properties like --num_executors via DBT-cli command: like this dbt run --select my_model --spark_conf "num-executors":5 ...
Gabber's user avatar
  • 7,229
0 votes
0 answers
200 views

Problems configuring LivyOperator in Airflow

For LivyOperator we set the following parameters: polling_interval=60 retries_num_timeout=100 We set it up according to this documentation: https://airflow.apache.org/docs/apache-airflow-providers-...
Павел Иванов's user avatar
2 votes
1 answer
236 views

How to setup virtual env or install a python library when I try to submit a PySpark job to Databricks from Airflow?

I need to submit a PySpark job to Airflow through LivyOperator. I see there are arguments to the LivyOperator's init method where users can pass in a list of Python files, but is there a way to do ...
WZH's user avatar
  • 434
0 votes
1 answer
252 views

how to switch Livy job log-level into DEBUG mode?

Like in spark we can add -Dlog4j.debug config , do we have any such equivalent in Airflow Livy operator. I have already browsed https://airflow.apache.org/docs/apache-airflow-providers-apache-livy/...
Priyanshu Sharma's user avatar
1 vote
1 answer
1k views

How to mock connection for airflow's Livy Operator using unittest.mock

@mock.patch.dict( "os.environ", AIRFLOW_CONN_LIVY_HOOK = "http://www.google.com", clear= True ) class TestLivyOperator(unittest.TestCase): def setUp(self): ...
Priyanshu Sharma's user avatar
1 vote
1 answer
773 views

Pyspark(via sparkmagic + livy) : There is insufficient memory for the Java Runtime Environment to continue

I'm using Sagemaker connecting to an EMR cluster via sparkmagic and livy, very frequently I get(at session startup, not running any code): > The code failed because of a fatal error: Session &...
Luis Leal's user avatar
  • 3,482
0 votes
0 answers
1k views

Livy logs location(in S3) for an EMR cluster(debuging Neither SparkSession nor HiveContext/SqlContext is available)

I'm using AWS SageMaker connected to an EMR cluster via Livy, with a "normal" session(default session config) the connection is created, and spark context works fine. but when adding spark....
Luis Leal's user avatar
  • 3,482
0 votes
1 answer
371 views

Apache Livy : Could not find or load main class org.apache.livy.server.LivyServer

I am trying to start Apache Livy 0.8.0 server on my windows 10 machine for spark 3.1.2 and hadoop 3.2.1. I am taking help from here.. I have successfully built apache livy using maven (I have attached ...
Mirza Asad's user avatar
1 vote
1 answer
590 views

Can't init session in Spark. How to debug "User capacity has reached its maximum limit."?

I'm trying to create a session in Apache Spark using the Livy rest API. It fails with the following error: User capacity has reached its maximum limit.. The user is running another spark job. I don't ...
neves's user avatar
  • 38k
2 votes
1 answer
4k views

Azure Synapse Spark LIVY_JOB_STATE_ERROR

i'm experimenting the following error when executing any cell in my notebook: LIVY_JOB_STATE_ERROR: Livy session has failed. Session state: Killed. Error code: LIVY_JOB_STATE_ERROR. [(my.synapse....
frammnm's user avatar
  • 537
2 votes
0 answers
630 views

Start PySpark in Jupyter notebook on EMR 6.5

I am trying to start a pyspark job using Amazon EMR Jupyter hub feature, as follow: And with following code: from pyspark import SparkSession spark = SparkSession \ .builder \ .appName("...
Gregoire's user avatar
  • 131
1 vote
0 answers
97 views

How to get the state of a remote job in Livy using Java API

Is it possible to monitor the state of an already running remote job in Livy with Java API? How can this be done? I looked over Livy Java API docs. A JobHandle would let me pool the state of the app. ...
oskarryn's user avatar
  • 343
0 votes
0 answers
263 views

Orchestration pyspark scripts in aws step functions

My spark job(we are processing 100 GB of data) taking one hour to complete.in step functions i am using lambda to submit our job through Livy!! and created one more lambda function to get the job ...
Siri Vali's user avatar
1 vote
0 answers
274 views

How can we connect to remote spark cluster via jupyterhub?

First of all my agenda is to be able to use spark codes inside jupyterhub. In other words I want to connect a remote spark cluster to jupyterhub. After searching about it I came up with two solutions:...
FarzanehTaheri's user avatar
1 vote
2 answers
881 views

Spark requests for more core than asked when calling POST livy batch api in azure synapse

I have an azure synapse spark cluster with 3 nodes of 4 vCores and 32 GB memory each. I am trying to submit a spark job using azure synapse Livy batch APIs. The request looks like this, curl --...
aman_kumar's user avatar
0 votes
1 answer
912 views

Livy on K8S, namespace restriction

I have spark (3.0.1), livy (0.8.0) and Jupyterhub (sparkmagic) running on K8S in specific namespace, Kubernetes master is used as a resource manager. When trying to create pyspark session in ...
Artyom Rebrov's user avatar
4 votes
0 answers
1k views

PySpark virtual environment archive on S3

I'm trying to deploy PySpark applications to an EMR cluster that have various, differing, third-party dependencies, and I am following this blog post, which describes a few approaches to packaging a ...
user4601931's user avatar
  • 5,239
0 votes
1 answer
541 views

I have trouble with using apache-livy, my interpreter doesn't work

I am currently trying to make a REST API server with apache livy. I started a livy-server and with my python code I used the livy api. Here's my python code that interacts with the livy-server import ...
cicada's user avatar
  • 19
0 votes
1 answer
348 views

How do I set up sparkmagic to work with DataProc through Livy?

I have a DataProc cluster running in GCP. I ran the Livy initialization script for it, and I can access the livy/sessions link through the gateway interface. I have the following set up for my ...
oneextrafact's user avatar
1 vote
1 answer
310 views

Databricks Notebook as Substitute for livy sessions endpoint

I want to execute a Databrikcs Notebook's code via Databricks API and get the output of notebook's code as response. Is it possible of is there any workaround for the same ? Is the same possible with ...
Shad Khan's user avatar
0 votes
0 answers
138 views

How to create a cached context in Spark using Apache Livy?

I would like to create a cached context using Apache Livy, such that while submitting spark jobs, it does not require creating a new context every time.
Abhisek Ray's user avatar
1 vote
0 answers
247 views

PySpark batch job's configuration submitted through Apache Livy have no effect

I submitted spark batch job through Livy to the remote cluster with the following request body. REQUEST_BODY = { 'file': '/spark/batch/job.py', 'conf': { 'spark.sql.legacy....
Korntewin Boonchuay's user avatar
2 votes
1 answer
829 views

SparkMagic PySpark3 session with Livy on Cloudera

I am trying to run Jupyterhub pyspark kernel session with python3 witn Livy running on Cloudera cluster. The spark session ends without any meaningful error, Livy logs have the following: 21/07/21 12:...
Artyom Rebrov's user avatar
0 votes
1 answer
752 views

AWS - Lambda cannot access Livy endpoint for EMR

For reference, livy is a rest endpoint used to pull data from a cluster. Within the same account, my lambda function always times out when attempting to access the livy endpoint using by EMR. Endpoint:...
Jeremy's user avatar
  • 5,425
2 votes
1 answer
366 views

How to include BigQuery Connector inside Dataproc using Livy

I'm trying to run my application using Livy that resides inside GCP Dataproc but I'm getting this: "Caused by: java.lang.ClassNotFoundException: bigquery.DefaultSource" I'm able to run ...
Celso Marques's user avatar
2 votes
1 answer
1k views

AWS MWAA - connection timed out when try do rest call to Livy

I created MWAA using public network option (version 2.0.2). Created a sample of airflow dag in which start emr with next properties: JOB_FLOW_OVERRIDES = { 'Name': 'demo-cluster-airflow', '...
Grish's user avatar
  • 331
2 votes
1 answer
1k views

Apache Livy 0.7.0 Failed to create Interactive session

While creating a new session using apache Livy 0.7.0 I am getting below error. I am also using zeppelin notebook(livy interpreter) to create the session. Using Scala version 2.12.10, Java HotSpot(TM) ...
Sushil Behera's user avatar
0 votes
2 answers
363 views

need help on submitting hudi delta streamer job via apache livy

I am little confused with how to pass the arguments as REST API JSON. Consider below spark submit command. spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.5.3,org.apache.spark:...
shiva nagesh's user avatar
0 votes
1 answer
1k views

spark job submitted via Livy throws GSSException: No valid credentials provided (Mechanism level: Failed to find any kerberos tgt)

I am trying to launch my spark batch job using livy. From the logs , i see that the start running but fails when it tries to access hive metastore with the following kerberos error: GSSException: No ...
smang's user avatar
  • 105
2 votes
1 answer
739 views

Jupyterhub pyspark3 on AWS EMR YARN Cluster

I'm running Jupyterhub with pyspark3 kernel on AWS EMR Cluster. As we might know Jupyterhub pyspark3 on EMR uses Livy session to run workloads on AWS EMR YARN scheduler. My question is about the ...
nael.fridhi's user avatar
2 votes
1 answer
1k views

Change Python version Livy uses in an EMR cluster

I am aware of Change Apache Livy's Python Version and How do i setup Pyspark in Python 3 with spark-env.sh.template. I also have seen the Livy documentation However, none of that works. Livy keeps ...
Clay's user avatar
  • 2,686
1 vote
1 answer
2k views

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails) livy 0.7.1 CDH 6.3.2(enable kerberos) in livy.conf livy.server.auth.type = kerberos livy.server....
ighack's user avatar
  • 31

1
2 3 4 5 6