Every day I have a hive job that compute some aggregations for each quarter of hour for two months of data. It's resulting in submitting something like 5760 jobs to Tez.
The job failed with the following error in the stderr :
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
The error could occurred after 2300 - 2500 tez jobs. Just before this error, There is a lot of the following logs in the Yarn Logs :
2015-12-10 21:53:35,286 INFO [TezChild] task.ContainerReporter: Sleeping for 200ms before retrying getTask again. Got null now. Next getTask sleep message after 2000ms
And the execution time of a single job is dramatically expending, from 20s to 100s.
I don't have any clue about my issue and I can't find anything else in yarn, Hadoop, hive, or tez logs (no exceptions, nothing marked as an error).
So I have two questions : Where can I find more information in logs or something else that could help me to resolve this issue ?
Currently we use :
- The latest version of Azure HDInsight 3.2
- Jobs are submitted to the cluster with the C# SDK
- Hive jobs use tez
Question 2 : I'm pretty sure that we do not do our aggregations in a good way. For each aggregation (i.e for each quarter), we should retrieve the precedent value of a row. I hoped to use the LAG function, but we could not pass a predicate for finding the precedent value (we need the precedent value that is not greater than the current value). So we could not find an other way than generate a query for each quarter that we need to compute. Is someone know how we can do that in a single hive query ?
Thanks in advance for any help, Best regards
tez.session.client.timeout.secs
andtez.session.am.dag.submit.timeout.secs
and so on.tez.session.client.timeout.secs
&tez.session.am.dag.submit.timeout.secs
but it does not change anything. I have also tried to decrease the container size for having more available containers. No changes