4

Where are the dataproc spark job logs located? I know there are logs from the driver under "Logging" section but what about the execution nodes? Also, where are the detailed steps that Spark is executing logged (I know I can see them in the Application Master)? I am attempting to debug a script that seems to hang and spark seems to freeze.

1 Answer 1

2

UPDATE in Q3 2022: This answer is outdated, see Dataproc YARN container logs location for the latest info.

The task logs are stored on each worker node under /tmp.

It is possible to collect them in one place via yarn log aggregation. Set these properties at cluster creation time (via --properties with yarn: prefix):

  • yarn.log-aggregation-enable=true
  • yarn.nodemanager.remote-app-log-dir=gs://${LOG_BUCKET}/logs
  • yarn.log-aggregation.retain-seconds=-1

Here's an article that discusses log management:

https://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/

2
  • which config file are these properties part of?
    – Alex
    Nov 20, 2017 at 19:52
  • as the property itself says, yarn-site.xml Sep 23, 2018 at 11:46

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.