19

I was trying to run a long running Spark Job. After few hours of execution, I get exception below :

Caused by: java.io.IOException: Failed to create local dir in /tmp/blockmgr-bb765fd4-361f-4ee4-a6ef-adc547d8d838/28 

Tried to get around it by checking:

  1. Permission issue in /tmp dir. The spark server is not running as root. but /tmp dir should be writable to all users.

  2. /tmp Dir has enough space.

1
  • Did you solve this issue?? Feb 1, 2019 at 2:41

3 Answers 3

4

Assuming that you are working with several nodes, you'll need to check every node participate in the spark operation (master/driver + slaves/nodes/workers).

Please confirm that each worker/node have enough disk space (especially check /tmp folder), and right permissions.

6
  • 2
    space is available on all nodes and /tmp is readable, writable. But still i get this issue in zeppelin.. Aug 8, 2017 at 9:31
  • Did you figure out what the problem was? I have the same one now. Apr 11, 2018 at 8:35
  • @AndreyCheptsov - did you confirm that you have enough disk space as described in my answer?
    – Yaron
    Apr 11, 2018 at 8:37
  • 1
    @Yaron yes, permissions and space were OK. What helped is a restart of the cluster. Apr 12, 2018 at 9:37
  • @Yaron Can you clarify what the "right permissions" are? I'm seeing this issue on AWS EMR (AWS managed spark cluster), and /tmp is actually an external file mount Feb 1, 2019 at 2:23
3

Edit: The answer below did not eventually solve my case. It's because some subfolders spark (or some of its dependencies) was able to create, yet not all of them. The frequent necessity of creation of such paths would make any project unviable. Therefore I ran Spark (PySpark in my case) as an Administrator, which solved the case. So in the end it is probably a permission issue afterall.


Original answer:
I solved the same problem I had on my local Windows machine (not a cluster). Since there was no problem with permissions, I created the dir that Spark was failing to create, i.e. I created the following folder as a local user and did not need to change any permissions on that folder.

C:\Users\<username>\AppData\Local\Temp\blockmgr-97439a5f-45b0-4257-a773-2b7650d17142
0

After verifying all the permissions and user access.

I got the same issue when building the components in Talend studio and it resolved by providing the correct "/" in spark scratch directory (temp directory) in spark Configuration tab. This is required when building the jar in windows and running in Linux cluster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.