291
questions
0
votes
0
answers
37
views
How can I read PySpark code from S3 buckets?
I have my Apache Livy setup and PySpark setup in AWS and I have kept my PySpark code in S3 bucket.
Here is my code to submit via Livy:
import requests
import json
livy_url_battches = "http://56....
0
votes
0
answers
15
views
How to check spark job on Amazon EMR console by job submit with lvy
I can successfully submit spark job through Apache livy to Amazon EMR.
But I did not see any steps on Amazon EMR console.
Though my spark job was completed and success, I could only check job state ...
0
votes
0
answers
30
views
AWS EMR Jupyterhub notebook run fails with error: Session isn't active
I noticed this question is asked a lot but the solution proposed in other threads do not work for me.
I have created an EMR cluster on AWS and I am running a quite time intensive code. The notebook ...
0
votes
1
answer
40
views
Can Spark SQL utilize previous results
I am curious if Spark can utilize previous results to help execution, for the example below:
Inside same session, I first create a view1 with some contents. Because there is no action, nothing is ...
0
votes
0
answers
80
views
How to use livy with external library
I try to use livy statement to submit a piece of scala code that reads lucene index and then read S3 files. As you can imagine, I try to use spark/livy to search data. Is that possible? How to include ...
0
votes
1
answer
49
views
Why Livy does not accept a large JSON args
Context: I have a C# API that sends an HTTP POST request to the /batches route of Livy, and Livy forwards the arguments to my Scala Spark driver. As far as I know, internally Livy runs the spark-...
0
votes
0
answers
49
views
facing error while creating spark session using apache livy ERROR RestSubmissionClient: Exception from the cluster
ERROR RestSubmissionClient: Exception from the cluster: java.nio.file.NoSuchFileException: spark-internal
Facing issue while creating spark session using Apache Livy.
Below are versions using.
apache-...
0
votes
0
answers
20
views
during high load on yarn cluster, spark session take more time to initialize
I observed that spark session take time to initialize when there is high load on cluster , can someone help me to understand why this happen, is spark maintain session pool somewhere and during high ...
0
votes
0
answers
34
views
Livy session to submit pyspark from HDFS
I am trying to run a simple spark sql basically to extract data from a hive table using Livy 0.8.0 using below code. Firewall is already enabled for Livy url.
Import requests
from requests_kerberos ...
0
votes
0
answers
83
views
Can not send livy rest api using proxy user into kerberized hadoop cluster
We want to use proxyuser in livy rest api for processing the data in kerberized hadoop cluster.
i.e. we want to impersonate user roslinda.statistik with livy user as below
curl -X POST -H "...
2
votes
1
answer
99
views
Apache Livy session showing Application id NULL
Whenever I try to fetch the status of the livy application which ran on a GCP Dataproc cluster, It always returned me null value, irrespective what I changed in the livy.conf, This is bugging me,
I ...
0
votes
0
answers
46
views
Accessing Dataproc Cluster through Apache Livy?
Can i access Dataproc cluster from Apache Livy instead of Dataproc API? If i can, Where I have to install my apache livy server?
Currently we need to access Dataproc cluster through apache livy ...
0
votes
0
answers
58
views
Python and Livy compatible version
I have a VM in which livy server is running, and livy version is 0.7.1 and spark version is 2.4.8 and I'm using python 3.7 version
Can I upgrade the my python version to >=3.8, will that be ...
0
votes
0
answers
223
views
Why Livy session is not found after YARN application succeed?
Using pylivy I create session and download resulting dataframe. The running code itself is not important:
with LivySession.create(self.LIVY_URL, kind=SessionKind.PYSPARK, requests_session=self....
-1
votes
1
answer
259
views
TypeError: required field "type_ignores" missing from Module Error Using Spark, Livy, Sparkmagic
where if I run my code as follows in separate cells line-by-line, there is no error like
using Jupyter notebook code block
import sys
print(sys.version)
3.8.9 ..
But, when i write code together ...
0
votes
0
answers
107
views
How to determine the practical character limit of a Livy request to submit Spark batch job sent from Airflow?
I am trying to figure out the upper limit (practical estimate is fine) of characters/bytes passed on as args to the LivyOperator in Airflow.
The operator eventually uses args(: Sequence[str | int | ...
0
votes
1
answer
175
views
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB is unauthorized Error?
I have kerberized Hadoop
I set this in core-site.xml
<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
...
0
votes
0
answers
872
views
EOL/Obsolete Software: Apache Log4j 1.X Detected
currently we are having Apache Log4j 1.X on one of our docker image which is reported as EOL and needs to update to Apache Log4j 2.X version but we are unable to find that package to upgrade it to ...
0
votes
1
answer
427
views
How to pass Spark executor parameters to DBT?
I am using DBT with spark-livy-adapter. And I want to pass spark-conf properties like --num_executors via DBT-cli command:
like this
dbt run --select my_model --spark_conf "num-executors":5
...
0
votes
0
answers
200
views
Problems configuring LivyOperator in Airflow
For LivyOperator we set the following parameters:
polling_interval=60
retries_num_timeout=100
We set it up according to this documentation: https://airflow.apache.org/docs/apache-airflow-providers-...
2
votes
1
answer
236
views
How to setup virtual env or install a python library when I try to submit a PySpark job to Databricks from Airflow?
I need to submit a PySpark job to Airflow through LivyOperator. I see there are arguments to the LivyOperator's init method where users can pass in a list of Python files, but is there a way to do ...
0
votes
1
answer
252
views
how to switch Livy job log-level into DEBUG mode?
Like in spark we can add -Dlog4j.debug config , do we have any such equivalent in Airflow Livy operator. I have already browsed https://airflow.apache.org/docs/apache-airflow-providers-apache-livy/...
1
vote
1
answer
1k
views
How to mock connection for airflow's Livy Operator using unittest.mock
@mock.patch.dict(
"os.environ",
AIRFLOW_CONN_LIVY_HOOK = "http://www.google.com",
clear= True
)
class TestLivyOperator(unittest.TestCase):
def setUp(self):
...
1
vote
1
answer
773
views
Pyspark(via sparkmagic + livy) : There is insufficient memory for the Java Runtime Environment to continue
I'm using Sagemaker connecting to an EMR cluster via sparkmagic and livy, very frequently I get(at session startup, not running any code):
> The code failed because of a fatal error: Session &...
0
votes
0
answers
1k
views
Livy logs location(in S3) for an EMR cluster(debuging Neither SparkSession nor HiveContext/SqlContext is available)
I'm using AWS SageMaker connected to an EMR cluster via Livy, with a "normal" session(default session config) the connection is created, and spark context works fine. but when adding
spark....
0
votes
1
answer
371
views
Apache Livy : Could not find or load main class org.apache.livy.server.LivyServer
I am trying to start Apache Livy 0.8.0 server on my windows 10 machine for spark 3.1.2 and hadoop 3.2.1. I am taking help from here.. I have successfully built apache livy using maven (I have attached ...
1
vote
1
answer
590
views
Can't init session in Spark. How to debug "User capacity has reached its maximum limit."?
I'm trying to create a session in Apache Spark using the Livy rest API. It fails with the following error: User capacity has reached its maximum limit..
The user is running another spark job. I don't ...
2
votes
1
answer
4k
views
Azure Synapse Spark LIVY_JOB_STATE_ERROR
i'm experimenting the following error when executing any cell in my notebook:
LIVY_JOB_STATE_ERROR: Livy session has failed. Session state: Killed. Error code: LIVY_JOB_STATE_ERROR. [(my.synapse....
2
votes
0
answers
630
views
Start PySpark in Jupyter notebook on EMR 6.5
I am trying to start a pyspark job using Amazon EMR Jupyter hub feature, as follow:
And with following code:
from pyspark import SparkSession
spark = SparkSession \
.builder \
.appName("...
1
vote
0
answers
97
views
How to get the state of a remote job in Livy using Java API
Is it possible to monitor the state of an already running remote job in Livy with Java API? How can this be done?
I looked over Livy Java API docs. A JobHandle would let me pool the state of the app. ...
0
votes
0
answers
263
views
Orchestration pyspark scripts in aws step functions
My spark job(we are processing 100 GB of data) taking one hour to complete.in step functions i am using lambda to submit our job through Livy!! and created one more lambda function to get the job ...
1
vote
0
answers
274
views
How can we connect to remote spark cluster via jupyterhub?
First of all my agenda is to be able to use spark codes inside jupyterhub. In other words I want to connect a remote spark cluster to jupyterhub. After searching about it I came up with two solutions:...
1
vote
2
answers
881
views
Spark requests for more core than asked when calling POST livy batch api in azure synapse
I have an azure synapse spark cluster with 3 nodes of 4 vCores and 32 GB memory each. I am trying to submit a spark job using azure synapse Livy batch APIs. The request looks like this,
curl --...
0
votes
1
answer
912
views
Livy on K8S, namespace restriction
I have spark (3.0.1), livy (0.8.0) and Jupyterhub (sparkmagic) running on K8S in specific namespace, Kubernetes master is used as a resource manager.
When trying to create pyspark session in ...
4
votes
0
answers
1k
views
PySpark virtual environment archive on S3
I'm trying to deploy PySpark applications to an EMR cluster that have various, differing, third-party dependencies, and I am following this blog post, which describes a few approaches to packaging a ...
0
votes
1
answer
541
views
I have trouble with using apache-livy, my interpreter doesn't work
I am currently trying to make a REST API server with apache livy. I started a livy-server and with my python code I used the livy api.
Here's my python code that interacts with the livy-server
import ...
0
votes
1
answer
348
views
How do I set up sparkmagic to work with DataProc through Livy?
I have a DataProc cluster running in GCP. I ran the Livy initialization script for it, and I can access the livy/sessions link through the gateway interface. I have the following set up for my ...
1
vote
1
answer
310
views
Databricks Notebook as Substitute for livy sessions endpoint
I want to execute a Databrikcs Notebook's code via Databricks API and get the output of notebook's code as response.
Is it possible of is there any workaround for the same ?
Is the same possible with ...
0
votes
0
answers
138
views
How to create a cached context in Spark using Apache Livy?
I would like to create a cached context using Apache Livy, such that while submitting spark jobs, it does not require creating a new context every time.
1
vote
0
answers
247
views
PySpark batch job's configuration submitted through Apache Livy have no effect
I submitted spark batch job through Livy to the remote cluster with the following request body.
REQUEST_BODY = {
'file': '/spark/batch/job.py',
'conf': {
'spark.sql.legacy....
2
votes
1
answer
829
views
SparkMagic PySpark3 session with Livy on Cloudera
I am trying to run Jupyterhub pyspark kernel session with python3 witn Livy running on Cloudera cluster.
The spark session ends without any meaningful error, Livy logs have the following:
21/07/21 12:...
0
votes
1
answer
752
views
AWS - Lambda cannot access Livy endpoint for EMR
For reference, livy is a rest endpoint used to pull data from a cluster. Within the same account, my lambda function always times out when attempting to access the livy endpoint using by EMR.
Endpoint:...
2
votes
1
answer
366
views
How to include BigQuery Connector inside Dataproc using Livy
I'm trying to run my application using Livy that resides inside GCP Dataproc but I'm getting this: "Caused by: java.lang.ClassNotFoundException: bigquery.DefaultSource"
I'm able to run ...
2
votes
1
answer
1k
views
AWS MWAA - connection timed out when try do rest call to Livy
I created MWAA using public network option (version 2.0.2).
Created a sample of airflow dag in which start emr with next properties:
JOB_FLOW_OVERRIDES = {
'Name': 'demo-cluster-airflow',
'...
2
votes
1
answer
1k
views
Apache Livy 0.7.0 Failed to create Interactive session
While creating a new session using apache Livy 0.7.0 I am getting below error.
I am also using zeppelin notebook(livy interpreter) to create the session.
Using Scala version 2.12.10, Java HotSpot(TM) ...
0
votes
2
answers
363
views
need help on submitting hudi delta streamer job via apache livy
I am little confused with how to pass the arguments as REST API JSON.
Consider below spark submit command.
spark-submit --packages org.apache.hudi:hudi-utilities-bundle_2.11:0.5.3,org.apache.spark:...
0
votes
1
answer
1k
views
spark job submitted via Livy throws GSSException: No valid credentials provided (Mechanism level: Failed to find any kerberos tgt)
I am trying to launch my spark batch job using livy. From the logs , i see that the start running but fails when it tries to access hive metastore with the following kerberos error:
GSSException: No ...
2
votes
1
answer
739
views
Jupyterhub pyspark3 on AWS EMR YARN Cluster
I'm running Jupyterhub with pyspark3 kernel on AWS EMR Cluster. As we might know Jupyterhub pyspark3 on EMR uses Livy session to run workloads on AWS EMR YARN scheduler. My question is about the ...
2
votes
1
answer
1k
views
Change Python version Livy uses in an EMR cluster
I am aware of Change Apache Livy's Python Version and How do i setup Pyspark in Python 3 with spark-env.sh.template.
I also have seen the Livy documentation
However, none of that works. Livy keeps ...
1
vote
1
answer
2k
views
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
livy 0.7.1
CDH 6.3.2(enable kerberos)
in livy.conf
livy.server.auth.type = kerberos
livy.server....