Questions tagged [oozie]
Oozie is a workflow/coordination system to manage Hadoop Map Reduce jobs
1,928
questions
0
votes
0
answers
8
views
oozie calling spark action java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found
I have this error after trying to call a spark action in oozie on kerberized cluster using CDP.
error from yarn log:
java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain ...
0
votes
0
answers
11
views
How to convert kerberos ticket into hadoop.auth cookie for the authenticating Http Hadoop Clients
My goal is to access the oozie Api's and deploy the jobs in the cluster. We have the kerberos authentication enabled to access the Oozie web console or the Oozie Api's. While I am trying to find the ...
0
votes
0
answers
10
views
Oozie coordinator setup
In my oozie coordinator , this is how I am defining the dependencies
<coordinator-app xmlns="uri:oozie:coordinator:0.1" timezone="UTC"
name="${appName}"
...
0
votes
0
answers
18
views
Some Spark Jobs failing when executing several with oozie
I have a oozie workflow with 17 concurrent spark actions. They execute the same program but with different input parameters.
The problem is that some of them is successful but another portion of them ...
0
votes
1
answer
12
views
Oozie coordinator app, how to configure action triggered by external data source?
I would like to run a job every time when a external data source is updated, for example, some government file is update, http://www.ic.gc.ca/folder/filename.zip. Is there way of doing it?
Please ...
0
votes
0
answers
17
views
FileNotFoundError for a json file on hdfs location when running oozie
I am trying to pass a dictionary of arguments to a script (a spark action) in an oozie.
The dictionary is in a .json file.
I pass the .json file as an argument in workflow.xml but deploying the ...
0
votes
0
answers
92
views
Oozie spark action involving spark sql fails if enableHiveSupport() is used
Using AWS EMR here.
Release label:emr-6.7.0
Hadoop distribution:Amazon 3.2.1
Applications:Spark 3.2.1, JupyterHub 1.4.1, Hue 4.10.0, Livy 0.7.1, Hive 3.1.3, Oozie 5.2.1
I'm trying to run a simple ...
0
votes
0
answers
21
views
oozie parse workflow.xml not as expected
I am new to oozie, and have a workflow.xml like below
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}&...
0
votes
0
answers
21
views
oozie dataset template uri using wildcard
I am trying to define a dependency to trigger my oozie workflow in my coordinator and my source file path is dynamic which cannot be defined before its presence, that's why i want to match my ...
0
votes
0
answers
16
views
Pyspark shell - Unable to receive output log file for errors
This is my script. I have edited this script for log filethrough some information on internet. Still i am unable to reveive log files for error. Anyone can help me solve this. This is my script.
from ...
0
votes
0
answers
84
views
How to schedule job on Oozie every exactly n day regardless of the month
I would like to schedule a job to run every n day where n is not 1 or 7. I know the syntax 0 0 */n * *, but when I check for the next execution dates, it always runs on 1st of the next month, ...
0
votes
0
answers
23
views
The sqoop import action job in oozie is still running
The sqoop import action job in oozie is still running.
What should I check?
oozie version:5.2.1
hadoop version:3.3.4
sqoop version:1.4.7
>>> Invoking Sqoop command line now >>>
...
0
votes
0
answers
36
views
Apache Oozie - How do I get a job's execution timestamp, and use it as an input for a PySpark workflow?
The current PySpark code I want to run takes a reference timestamp (let's call it "processing_date"), and uses it (via relativedelta) to set past dates needed for data acquisition and ...
0
votes
0
answers
20
views
Propagate ErrorCode from a specific Oozie action in subworkflow to main workflow
An action ends with a E1111 ErrorCode in an Oozie subworkflow. The main workflow (which launches the subworkflow) shows with that the subworkflow is in ERROR but the ErrorCode og the subworkflow is ...
-1
votes
1
answer
30
views
How completely remove user:admin filter in hue UI?
When I login to hue or refresh page I got filter with user:admin selected:
Is there a way to modify hue configs and remove this default filter? (It is very annoyuing to remove it on every page ...
0
votes
1
answer
75
views
Not able to run(schedule) oozie example map reduce job || java.net.ConnectException
I am using Hadoop 2.6.0 and oozie 5.2.0 version.
Trying to run example Map reduce job in oozie but getting below error.
hadoop1@ip-172-31-84-37:/usr/local/oozie-5.2.0/examples/target/examples/apps/map-...
0
votes
0
answers
26
views
How refer system variable in job.properties for oozie workflow?
Oozie has job.properties file for defining properties and variables. Is it possible to use ssytem variables in this file?
0
votes
0
answers
294
views
Getting error: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
I have hit an error when trying to run with Impala script. Below is the error codes, where was the issue from?
INFO : Kill Command = /var/opt/teradata/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p4155....
0
votes
0
answers
23
views
Oozie Workflow moving multiple files while changing their name
I'm relatively new to using oozie workflows and coordinators and I was running into a problem.
As part of the workflow we wanna archive files with the following script:
<mkdir path="${archive}&...
0
votes
0
answers
27
views
How to define retry in oozie ui?
I have some schedulers in OOZIE through hue and some of them some times fail. However when I run them manually after, they end successfully. Is there any way to put retry policy in my WFs?
0
votes
1
answer
57
views
Use class variables inside Oozie workflow
I have an oozie workflow that has the following format:
<workflow-app xmlns="uri:oozie:workflow:0.5" name="${componente}">
...
<start to="...
0
votes
0
answers
22
views
Hue Oozie PySpark job: Reduce data being ported to stdout?
I'm a user on a Hue platform and am creating an Oozie job using PySpark [this is through a Hue workflow GUI]. When I submit, a large amount of information about dispatchers and tasks are ported to ...
0
votes
1
answer
83
views
How to find what arguments are passed to a Scala main class from oozie spark action
I'm trying to figure out if I can access the name parameter from <name>[SPARK JOB NAME]</name> inside the Scala class, when running a spark application.
Spark Action: https://oozie.apache....
0
votes
0
answers
77
views
I need to configure Oozie coordinator dataset for last previous day
How can I schedule oozie coordinator for previous days from the latest current day.
For Example:
If I need to pick the payload path the folder format of dataset will be like below
ts=202205190300/
ts=...
0
votes
0
answers
50
views
Connecting to an Oozie Server
I'm trying to connect to Oozie on localhost and the standard command shows connection timeout. The server says it's running when I write
oozied.sh run
When I try accessing http://localhost:11000 it ...
0
votes
0
answers
28
views
Oozie integration with livy
I am trying to submit a spark job using oozie scheduler. I need to use livy for this, so that the job will always be submitted to the master node of EMR. Is there any option to do this on hue UI? ...
0
votes
0
answers
131
views
Apache Airflow/Azkaban workflow Schedulers compatibility with Hadoop MRv1
I'm working on a project that relies on Hadoop but MRv1 architecture (Hadoop-1.1.2). I tried oozie scheduler for creating workflows(mapred) but gave up eventually, cause it is a nightmare to configure ...
3
votes
0
answers
90
views
Cannot execute workflows in Hue Load Balancer
I'm trying to submit a workflow from the Hue Load Balancer and I'm having the following error:
"E0307: Runtime error [XML file [hdfs/path/to/workflow.xml] does not exist and app definition cannot ...
0
votes
0
answers
86
views
How to get application id from oozie job id
Is there a way to find out the applicationid using the oozie job id in single command?
I want to display the application id of the failed jobs for an oozie id.
oozie job -info
Can any option be used ...
0
votes
0
answers
44
views
Sqoop command not found when running through Oozie
When I am running Sqoop script in CLI, it is running fine without any issue. But when run it using Oozie, it failed with Sqoop command not found. It seems sqoop is not installed in other data nodes. ...
0
votes
0
answers
139
views
On HDinsight 4.0, how can i save/update table on hive and this table be readable for oozie/spark/jupyter
Today we have this scenario:
Cluster Azure HDInsight 4.0
Running Workflow on Oozie
On this version, spark and hive do not share metadata anymore
We came from HDInsight 3.6, to work with this change, ...
0
votes
1
answer
25
views
Executing multiple mapred jobs that are executed in a sequence with a delay
I wanted to know if it was possible to execute/or submit a mapred job to be executed in the hadoop cluster after a given delay from the job before. To explain clearly, jobs are submitted in a sequence ...
0
votes
1
answer
106
views
Packaging shell action files with Oozie, retaining original directory structure
I have a PySpark application I would like to schedule with Oozie, using the shell action.
My submit-application.sh script simply initializes a Python virtualenv (present on all worker nodes) and calls ...
0
votes
1
answer
399
views
Snowflake cancel query when ABORT_DETACHED_QUERY=true on session level
I am using Snowflake JDBC to execute multi-statement scripts on Snowflake. My application is started by a Oozie job on a hadoop cluster(in migration phase). The requirement here is, when Oozie job is ...
0
votes
1
answer
152
views
Oozie coordinator get day of the week
I am trying to create a condition in my Oozie workflow, where an action should be executed only on mondays (at the end of the workflow).
So far I added a decision node in the workflow, and the current ...
0
votes
1
answer
176
views
Running Python Script in OOzie with special libraries without install them to server
I want to run Python script in OOzie workflow with special libraries. But, I want to run this script without installing these special libraries to Hadoop nodes. I tried to run with virtualenv but ...
0
votes
0
answers
53
views
Oozie workflow fails to execute hive query using UDF with com.google.i18n.phonenumbers.PhoneNumberUtil.parse()
I have created a .jar file that utilizes many functions from google.i18n.phonenumbers java package. The jar file correctly loads the necessary external libraries from com.googlecode.libphonenumber....
0
votes
0
answers
462
views
Getting "Delegation Token can be issued only with kerberos or web authentication" executing Hive query in shell script
I am trying to connect Hive and execute query in shell script triggered by oozie, the server is kerberos enabled. I am passing hive credentials in workflow but I am still getting the error output ...
0
votes
0
answers
154
views
Get last day of month :: Expression language evaluation error :${coord:dateOffset(${coord:endOfMonths(0)}, -1, 'DAY')}:
I want to get last day of previous month and don't understand where is the error
<value>${coord:formatTime(${coord:dateOffset(${coord:endOfMonths(0)}, -1, 'DAY')}, 'yyyy-MM-dd')}</value>
-1
votes
1
answer
159
views
Is there a way to install apache oozie in windows 10
How I can install apache oozie in windows 10, not vm
I could not find any documents.
Thanks
1
vote
1
answer
241
views
Sqoop Import command throwing error but it is working from terminal
Below command is working fine from terminal but when I extecute it from Oozie it is throwing error.
bash
sqoop import --connect jdbc:mysql://172.16.16.128:3306/employees -- username=hive --password=...
0
votes
0
answers
271
views
Delay in application submission from Oozie and Yarn
We are running a Oozie workflow which has Shell action and a Spark action which means a shell script and a Spark job which runs in sequence.
Running single workflow:
Total: 3 mins
Shell action: 50 ...
-1
votes
1
answer
255
views
Airflow Map-Reduce Operator
I am looking for an airflow map reduce operator. I use OOZIE map reduce action now, and trying to convert to Airflow. I have looked under
https://airflow.apache.org/docs/apache-airflow/1.10.14/_api/...
0
votes
1
answer
122
views
Oozie: why error log shows in mapreduce job, not in Spark job?
I submitted a oozie workflow that is a shell action, it calls spark-submit to run a Spring boot application which is a jar file. It runs on yarn in client mode.
However, I found that the all Spring ...
1
vote
0
answers
136
views
Oozie workflow example failing despite running oozie-setup.sh sharelib with EJ001
I'm trying to run the examples documented at https://oozie.apache.org/docs/5.2.1/DG_Examples.html
When I run the first example it fails with EJ001:
root@50e1f53d5ded:/opt/oozie# oozie job -oozie http:/...
0
votes
1
answer
68
views
can OOZIE Map-Reduce job save data in parquet format?
I have a Map Reduce job that uses OOZIE workflow xml and writes output in sequence file format(org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat), is there something like this for saving ...
1
vote
0
answers
359
views
Spark with Hive warehouse connector within oozie
I'm using spark with hive warehouse connector. When i start a spark-submit from the edge node using a shell script it works perfectly, but when i start an oozie job that runs the same script it ...
0
votes
3
answers
440
views
failure to login: for principal: jztwk javax.security.auth.login.LoginException: Unable to obtain password from user
I use CDH 6.3.2
I sumbit oozie job with spark,but I get a error
..................................
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
Caused by: org.apache....
1
vote
3
answers
613
views
in oozei with spark java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
I use CDH 6.3.2
hadoop is HA
I make a workflow with spark in hue
run this workflow I get a error
Failing Oozie Launcher, java.net.UnknownHostException: nameservice1
java.lang.IllegalArgumentException: ...
0
votes
1
answer
142
views
Oozie's job: yarn returns Error starting action [hive-4548]
There is a cluster with Cloudera including Hue. My need is the task for scheduler which send HQL-request to Hive.
I'm trying to do task for oozie by web-constructor integrated in Hue.
My HQL request'...