Questions tagged [oozie]

Oozie is a workflow/coordination system to manage Hadoop Map Reduce jobs

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

oozie calling spark action java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

I have this error after trying to call a spark action in oozie on kerberized cluster using CDP. error from yarn log: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain ...
  • 1,528
0 votes
0 answers
11 views

How to convert kerberos ticket into hadoop.auth cookie for the authenticating Http Hadoop Clients

My goal is to access the oozie Api's and deploy the jobs in the cluster. We have the kerberos authentication enabled to access the Oozie web console or the Oozie Api's. While I am trying to find the ...
  • 117
0 votes
0 answers
10 views

Oozie coordinator setup

In my oozie coordinator , this is how I am defining the dependencies <coordinator-app xmlns="uri:oozie:coordinator:0.1" timezone="UTC" name="${appName}" ...
  • 1,561
0 votes
0 answers
18 views

Some Spark Jobs failing when executing several with oozie

I have a oozie workflow with 17 concurrent spark actions. They execute the same program but with different input parameters. The problem is that some of them is successful but another portion of them ...
  • 43
0 votes
1 answer
12 views

Oozie coordinator app, how to configure action triggered by external data source?

I would like to run a job every time when a external data source is updated, for example, some government file is update, http://www.ic.gc.ca/folder/filename.zip. Is there way of doing it? Please ...
  • 1
0 votes
0 answers
17 views

FileNotFoundError for a json file on hdfs location when running oozie

I am trying to pass a dictionary of arguments to a script (a spark action) in an oozie. The dictionary is in a .json file. I pass the .json file as an argument in workflow.xml but deploying the ...
0 votes
0 answers
92 views

Oozie spark action involving spark sql fails if enableHiveSupport() is used

Using AWS EMR here. Release label:emr-6.7.0 Hadoop distribution:Amazon 3.2.1 Applications:Spark 3.2.1, JupyterHub 1.4.1, Hue 4.10.0, Livy 0.7.1, Hive 3.1.3, Oozie 5.2.1 I'm trying to run a simple ...
0 votes
0 answers
21 views

oozie parse workflow.xml not as expected

I am new to oozie, and have a workflow.xml like below <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}&...
  • 21
0 votes
0 answers
21 views

oozie dataset template uri using wildcard

I am trying to define a dependency to trigger my oozie workflow in my coordinator and my source file path is dynamic which cannot be defined before its presence, that's why i want to match my ...
  • 526
0 votes
0 answers
16 views

Pyspark shell - Unable to receive output log file for errors

This is my script. I have edited this script for log filethrough some information on internet. Still i am unable to reveive log files for error. Anyone can help me solve this. This is my script. from ...
0 votes
0 answers
84 views

How to schedule job on Oozie every exactly n day regardless of the month

I would like to schedule a job to run every n day where n is not 1 or 7. I know the syntax 0 0 */n * *, but when I check for the next execution dates, it always runs on 1st of the next month, ...
  • 501
0 votes
0 answers
23 views

The sqoop import action job in oozie is still running

The sqoop import action job in oozie is still running. What should I check? oozie version:5.2.1 hadoop version:3.3.4 sqoop version:1.4.7 >>> Invoking Sqoop command line now >>> ...
  • 3
0 votes
0 answers
36 views

Apache Oozie - How do I get a job's execution timestamp, and use it as an input for a PySpark workflow?

The current PySpark code I want to run takes a reference timestamp (let's call it "processing_date"), and uses it (via relativedelta) to set past dates needed for data acquisition and ...
0 votes
0 answers
20 views

Propagate ErrorCode from a specific Oozie action in subworkflow to main workflow

An action ends with a E1111 ErrorCode in an Oozie subworkflow. The main workflow (which launches the subworkflow) shows with that the subworkflow is in ERROR but the ErrorCode og the subworkflow is ...
  • 843
-1 votes
1 answer
30 views

How completely remove user:admin filter in hue UI?

When I login to hue or refresh page I got filter with user:admin selected: Is there a way to modify hue configs and remove this default filter? (It is very annoyuing to remove it on every page ...
  • 30.4k
0 votes
1 answer
75 views

Not able to run(schedule) oozie example map reduce job || java.net.ConnectException

I am using Hadoop 2.6.0 and oozie 5.2.0 version. Trying to run example Map reduce job in oozie but getting below error. hadoop1@ip-172-31-84-37:/usr/local/oozie-5.2.0/examples/target/examples/apps/map-...
  • 1
0 votes
0 answers
26 views

How refer system variable in job.properties for oozie workflow?

Oozie has job.properties file for defining properties and variables. Is it possible to use ssytem variables in this file?
  • 30.4k
0 votes
0 answers
294 views

Getting error: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

I have hit an error when trying to run with Impala script. Below is the error codes, where was the issue from? INFO : Kill Command = /var/opt/teradata/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p4155....
0 votes
0 answers
23 views

Oozie Workflow moving multiple files while changing their name

I'm relatively new to using oozie workflows and coordinators and I was running into a problem. As part of the workflow we wanna archive files with the following script: <mkdir path="${archive}&...
0 votes
0 answers
27 views

How to define retry in oozie ui?

I have some schedulers in OOZIE through hue and some of them some times fail. However when I run them manually after, they end successfully. Is there any way to put retry policy in my WFs?
  • 765
0 votes
1 answer
57 views

Use class variables inside Oozie workflow

I have an oozie workflow that has the following format: <workflow-app xmlns="uri:oozie:workflow:0.5" name="${componente}"> ... <start to="...
  • 841
0 votes
0 answers
22 views

Hue Oozie PySpark job: Reduce data being ported to stdout?

I'm a user on a Hue platform and am creating an Oozie job using PySpark [this is through a Hue workflow GUI]. When I submit, a large amount of information about dispatchers and tasks are ported to ...
  • 3,412
0 votes
1 answer
83 views

How to find what arguments are passed to a Scala main class from oozie spark action

I'm trying to figure out if I can access the name parameter from <name>[SPARK JOB NAME]</name> inside the Scala class, when running a spark application. Spark Action: https://oozie.apache....
  • 1
0 votes
0 answers
77 views

I need to configure Oozie coordinator dataset for last previous day

How can I schedule oozie coordinator for previous days from the latest current day. For Example: If I need to pick the payload path the folder format of dataset will be like below ts=202205190300/ ts=...
  • 1
0 votes
0 answers
50 views

Connecting to an Oozie Server

I'm trying to connect to Oozie on localhost and the standard command shows connection timeout. The server says it's running when I write oozied.sh run When I try accessing http://localhost:11000 it ...
0 votes
0 answers
28 views

Oozie integration with livy

I am trying to submit a spark job using oozie scheduler. I need to use livy for this, so that the job will always be submitted to the master node of EMR. Is there any option to do this on hue UI? ...
0 votes
0 answers
131 views

Apache Airflow/Azkaban workflow Schedulers compatibility with Hadoop MRv1

I'm working on a project that relies on Hadoop but MRv1 architecture (Hadoop-1.1.2). I tried oozie scheduler for creating workflows(mapred) but gave up eventually, cause it is a nightmare to configure ...
  • 3
3 votes
0 answers
90 views

Cannot execute workflows in Hue Load Balancer

I'm trying to submit a workflow from the Hue Load Balancer and I'm having the following error: "E0307: Runtime error [XML file [hdfs/path/to/workflow.xml] does not exist and app definition cannot ...
0 votes
0 answers
86 views

How to get application id from oozie job id

Is there a way to find out the applicationid using the oozie job id in single command? I want to display the application id of the failed jobs for an oozie id. oozie job -info Can any option be used ...
0 votes
0 answers
44 views

Sqoop command not found when running through Oozie

When I am running Sqoop script in CLI, it is running fine without any issue. But when run it using Oozie, it failed with Sqoop command not found. It seems sqoop is not installed in other data nodes. ...
0 votes
0 answers
139 views

On HDinsight 4.0, how can i save/update table on hive and this table be readable for oozie/spark/jupyter

Today we have this scenario: Cluster Azure HDInsight 4.0 Running Workflow on Oozie On this version, spark and hive do not share metadata anymore We came from HDInsight 3.6, to work with this change, ...
0 votes
1 answer
25 views

Executing multiple mapred jobs that are executed in a sequence with a delay

I wanted to know if it was possible to execute/or submit a mapred job to be executed in the hadoop cluster after a given delay from the job before. To explain clearly, jobs are submitted in a sequence ...
  • 3
0 votes
1 answer
106 views

Packaging shell action files with Oozie, retaining original directory structure

I have a PySpark application I would like to schedule with Oozie, using the shell action. My submit-application.sh script simply initializes a Python virtualenv (present on all worker nodes) and calls ...
0 votes
1 answer
399 views

Snowflake cancel query when ABORT_DETACHED_QUERY=true on session level

I am using Snowflake JDBC to execute multi-statement scripts on Snowflake. My application is started by a Oozie job on a hadoop cluster(in migration phase). The requirement here is, when Oozie job is ...
  • 144
0 votes
1 answer
152 views

Oozie coordinator get day of the week

I am trying to create a condition in my Oozie workflow, where an action should be executed only on mondays (at the end of the workflow). So far I added a decision node in the workflow, and the current ...
  • 11
0 votes
1 answer
176 views

Running Python Script in OOzie with special libraries without install them to server

I want to run Python script in OOzie workflow with special libraries. But, I want to run this script without installing these special libraries to Hadoop nodes. I tried to run with virtualenv but ...
0 votes
0 answers
53 views

Oozie workflow fails to execute hive query using UDF with com.google.i18n.phonenumbers.PhoneNumberUtil.parse()

I have created a .jar file that utilizes many functions from google.i18n.phonenumbers java package. The jar file correctly loads the necessary external libraries from com.googlecode.libphonenumber....
  • 1,232
0 votes
0 answers
462 views

Getting "Delegation Token can be issued only with kerberos or web authentication" executing Hive query in shell script

I am trying to connect Hive and execute query in shell script triggered by oozie, the server is kerberos enabled. I am passing hive credentials in workflow but I am still getting the error output ...
0 votes
0 answers
154 views

Get last day of month :: Expression language evaluation error :${coord:dateOffset(${coord:endOfMonths(0)}, -1, 'DAY')}:

I want to get last day of previous month and don't understand where is the error <value>${coord:formatTime(${coord:dateOffset(${coord:endOfMonths(0)}, -1, 'DAY')}, 'yyyy-MM-dd')}</value>
-1 votes
1 answer
159 views

Is there a way to install apache oozie in windows 10

How I can install apache oozie in windows 10, not vm I could not find any documents. Thanks
1 vote
1 answer
241 views

Sqoop Import command throwing error but it is working from terminal

Below command is working fine from terminal but when I extecute it from Oozie it is throwing error. bash sqoop import --connect jdbc:mysql://172.16.16.128:3306/employees -- username=hive --password=...
0 votes
0 answers
271 views

Delay in application submission from Oozie and Yarn

We are running a Oozie workflow which has Shell action and a Spark action which means a shell script and a Spark job which runs in sequence. Running single workflow: Total: 3 mins Shell action: 50 ...
  • 1
-1 votes
1 answer
255 views

Airflow Map-Reduce Operator

I am looking for an airflow map reduce operator. I use OOZIE map reduce action now, and trying to convert to Airflow. I have looked under https://airflow.apache.org/docs/apache-airflow/1.10.14/_api/...
  • 341
0 votes
1 answer
122 views

Oozie: why error log shows in mapreduce job, not in Spark job?

I submitted a oozie workflow that is a shell action, it calls spark-submit to run a Spring boot application which is a jar file. It runs on yarn in client mode. However, I found that the all Spring ...
  • 1,844
1 vote
0 answers
136 views

Oozie workflow example failing despite running oozie-setup.sh sharelib with EJ001

I'm trying to run the examples documented at https://oozie.apache.org/docs/5.2.1/DG_Examples.html When I run the first example it fails with EJ001: root@50e1f53d5ded:/opt/oozie# oozie job -oozie http:/...
  • 11
0 votes
1 answer
68 views

can OOZIE Map-Reduce job save data in parquet format?

I have a Map Reduce job that uses OOZIE workflow xml and writes output in sequence file format(org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat), is there something like this for saving ...
  • 341
1 vote
0 answers
359 views

Spark with Hive warehouse connector within oozie

I'm using spark with hive warehouse connector. When i start a spark-submit from the edge node using a shell script it works perfectly, but when i start an oozie job that runs the same script it ...
0 votes
3 answers
440 views

failure to login: for principal: jztwk javax.security.auth.login.LoginException: Unable to obtain password from user

I use CDH 6.3.2 I sumbit oozie job with spark,but I get a error .................................. at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141) Caused by: org.apache....
  • 31
1 vote
3 answers
613 views

in oozei with spark java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1

I use CDH 6.3.2 hadoop is HA I make a workflow with spark in hue run this workflow I get a error Failing Oozie Launcher, java.net.UnknownHostException: nameservice1 java.lang.IllegalArgumentException: ...
  • 31
0 votes
1 answer
142 views

Oozie's job: yarn returns Error starting action [hive-4548]

There is a cluster with Cloudera including Hue. My need is the task for scheduler which send HQL-request to Hive. I'm trying to do task for oozie by web-constructor integrated in Hue. My HQL request'...

1
2 3 4 5
39