57

I am running a Python script that is causing the above error. The unusual thing is this script is running on a different machine and is having no problems.

The difference is that on the machine that is causing the problems I am writing to an external hard drive. To make things even weirder this script has run on the problem machine and already written over 30,000 files.

Some relevant information (The code that is causing the error):

nPage = 0
while nPage != -1:
    for d in data:
        if len(d.contents) > 1:
            if '<script' in str(d.contents):
                l = str(d.contents[1])
                start = l.find('http://')
                end = l.find('>',start)
                out = get_records.openURL(l[start:end])
                print COUNT

                with open('../results/'+str(COUNT)+'.html','w') as f:
                    f.write(out)
                COUNT += 1

    nPage = nextPage(mOut,False)

The directory I'm writing to:

10:32@lorax:~/econ/estc/bin$ ll ../
total 56
drwxr-xr-x 3 boincuser boincuser  4096 2011-07-31 14:29 ./
drwxr-xr-x 3 boincuser boincuser  4096 2011-07-31 14:20 ../
drwxr-xr-x 2 boincuser boincuser  4096 2011-08-09 10:38 bin/
lrwxrwxrwx 1 boincuser boincuser    47 2011-07-31 14:21 results -> /media/cavalry/server_backup/econ/estc/results//
-rw-r--r-- 1 boincuser boincuser 44759 2011-08-09 10:32 test.html

Proof there is enough space:

10:38@lorax:~/econ/estc/bin$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.0G  5.3G  3.3G  63% /
none                  495M  348K  495M   1% /dev
none                  500M  164K  500M   1% /dev/shm
none                  500M  340K  500M   1% /var/run
none                  500M     0  500M   0% /var/lock
none                  9.0G  5.3G  3.3G  63% /var/lib/ureadahead/debugfs
/dev/sdc10            466G  223G  244G  48% /media/cavalry

Some things I have tried:

  • Changing the path of the write to the direct location instead of going through the link
  • Rebooting the machine
  • Unmounting and re-mounting the drive
7
  • What kind of file system is /media/cavalry? Maybe you could post the relevant line from /proc/mounts. Are you simply running out of inodes? Commented Aug 9, 2011 at 14:48
  • 4
    It would be helpful if you included full traceback (on which line the error occurred?), filesystem types and free inode count (df -i). In this case, I suspect the filesystem is vfat and you've exceeded the maximum number of files in a directory. Commented Aug 9, 2011 at 14:53
  • @Rosh - +1 - see that the filename is 32766 - that number sets off alarms :) btw, you should move your comment to answers
    – KevinDTimm
    Commented Aug 9, 2011 at 15:04
  • 1
    32766 + "." + ".." for 32768 directory entries. Hmm...
    – wberry
    Commented Aug 9, 2011 at 15:53
  • @Rosh Yes there are no more inodes. Interesting. Rosh if you post your comment to an answer I can list that as the choice. But I'm also wondering what I can do to rectify this and where I can get some more information on inodes in the vfat system. Thanks!
    – josh
    Commented Aug 9, 2011 at 16:29

10 Answers 10

72

The ENOSPC ("No space left on device") error will be triggered in any situation in which the data or the metadata associated with an I/O operation can't be written down anywhere because of lack of space. This doesn't always mean disk space – it could mean physical disk space, logical space (e.g. maximum file length), space in a certain data structure or address space. For example you can get it if there isn't space in the directory table (vfat) or there aren't any inodes left. It roughly means “I can't find where to write this down”.

Particularly in Python, this can happen on any write I/O operation. It can happen during f.write, but it can also happen on open, on f.flush and even on f.close. Where it happened provides a vital clue for the reason that it did – if it happened on open there wasn't enough space to write the metadata for the entry, if it happened during f.write, f.flush or f.close there wasn't enough disk space left or you've exceeded the maximum file size.

If the filesystem in the given directory is vfat you'd hit the maximum file limit at about the same time that you did. The limit is supposed to be 2^16 directory entries, but if I recall correctly some other factors can affect it (e.g. some files require more than one entry).

It would be best to avoid creating so many files in a directory. Few filesystems handle so many directory entries with ease. Unless you're certain that your filesystem deals well with many files in a directory, you can consider another strategy (e.g. create more directories).

P.S. Also do not trust the remaining disk space – some file systems reserve some space for root and others miscalculate the free space and give you a number that just isn't true.

6
  • First, this is happening on "open" line. Second, I mean to post this with my last comment:Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdc10 0 0 0 - /media/cavalry. So, it looks like I don't have any Inodes but also that I'm not using any. As for making more directories wouldn't that kind of defeat the point? Let's say my tree is /estc/results30000/ and estc/results60000/. If I put the first 30000 in results30000 and the next 30000 in results 60000 the estc/ directory will still have too many files. I am also confident on the free space
    – josh
    Commented Aug 9, 2011 at 16:49
  • 1
    I received the same error during a file write operation (download) in a Python script. The inodes as well as disk space were very much available. I had managed to run the Python script successfully on another machine and when I manually copied it to the first machine, the copy was successful. I don't understand how that worked if the issue was anything related to exceeding maximum file size, max directory/file count, or any other physical disk constraints. Any more information would be of so much help as I might encounter this issue again.
    – gvas
    Commented Sep 9, 2016 at 2:50
  • 2
    I have a similar error, using Python 3, Jupyter Notebooks, sklearn, RandomForestRegressor(n_jobs=-2). I believe it was because a function was using pickle a lot, but not sure about the real cause. Following this answer I wrote %env JOBLIB_TEMP_FOLDER=/tmp in the Python notebook and everythin fine. I am not sure if it is useful here or if someone can expand. Commented Aug 23, 2018 at 15:18
  • This is an yeasrs old answer, but I'll try to ask regardless (don't wanna open a duplicate question). I got the same error during np.save() in a setup tha loops over a bunch o cases. The error ocurred after about 1.5k iterations and there were 150GB left in disc, so I don't know what to make out of it.
    – Puff
    Commented Feb 25, 2021 at 17:42
  • I want to add another potential cause of ENOSPC that I ran into. I work on managed Linux instances, and there is a quota with a hard limit on the 'home' directory. Conda uses ~/.conda for storing packages. Eventually, I was unable to use 'conda create ...', as it would give the 'no space left on device' message, but there was no indication of what was going wrong. Disk space, inodes, etc. all looked fine. I only figured it out when my ~/.bashrc got truncated when I tried 'conda init'. Clearing out ~/.conda packages fixed it. Commented Mar 12, 2021 at 16:15
24

Try to delete the temp files

rm -r /tmp/
4
  • 35
    Dangerous use of cd followed by rm Commented Oct 18, 2017 at 16:39
  • 11
    This answer is underappreciated- other answers are essentially saying "Too much garbage? Get a bigger trash can! While this answer is taking out the trash. Commented Feb 23, 2018 at 18:12
  • 1
    is there a way to say 'y' yes to all questions asking if I'm sure?
    – Akin Hwan
    Commented Jun 25, 2019 at 14:41
  • This is the Best among all answers out there. Note : It is best practise to give yes/no y or n to each Temp directory to avoid deleting other users files/directory in ubuntu ! Commented Aug 7, 2019 at 7:27
10

It turns out the best solution for me here was to just reformat the drive. Once reformatted all these problems were no longer problems.

4
  • 5
    I had the same question. Would be nice if people left an explanation with their downvotes.
    – josh
    Commented Feb 14, 2017 at 16:19
  • 8
    The cause of the problem here seems to have been inodes / directory entries... (And it seems to have been vfat specific) That is not obvious in the question (without the comments) (and this answer mainly provides a workaround, which is not practical in many cases...) (You also did not specify whether it was formatted with vfat again, or a different filesystem...) Commented Oct 19, 2017 at 10:36
  • 3
    You cannot always reformat. Reformating is always an obvious solution. Did you not need to keep the data on your device??? That is a reason to downvote. Commented Jan 26, 2022 at 4:48
  • 1
    @josh Please explain how you "reformatted" the drive
    – DialFrost
    Commented Aug 13, 2022 at 4:04
8
  1. Show where memory is allocated sudo du -x -h / | sort -h | tail -40
  2. Delete from either your /tmp or /home/user_name/.cache folder if these are taking up a lot of memory. You can do this by running sudo rm -R /path/to/folder

Step 2 outlines fairly common folders to delete from (/tmp and /home/user_name/.cache). If you get back other results when running the first command showing you have lots of memory being used elsewhere, I advise being a bit more cautious when deleting from those locations.

0
5

In my case, when I run df -i it shows me that my number of inodes are full and then I have to delete some of the small files or folder. Otherwise it will not allow us to create files or folders once inodes get full.

All you have to do is delete files or folder that has not taken up full space but is responsible for filling inodes.

4

run "export TEMPDIR=/someDir" where some dir is a valid directory other than /tmp. Run this on prompt before running your python command. In my case it is "pip install rasa[spacy]" which was earlier failing.

The export command allows you to temporarily use the specified dir as temp dir.

1
1

Minimal reproducible error example with too many files in a single directory

The approximate maximum number of files per directory for different filesystems can be seen at: How many files can I put in a directory?

The following example eventually blows up giving us a concrete estimate to the size limit:

#!/usr/bin/env python

import os
import shutil

tmpdir = 'tmp'
if os.path.isdir(tmpdir):
    shutil.rmtree(tmpdir)
os.mkdir(tmpdir)
for i in range(10000000):
    print(i)
    with open(os.path.join(tmpdir, f'{i:064}'), 'w') as f:
        pass

and always blows up on my test system at:

5590508
Traceback (most recent call last):
  File "/home/ciro/test/tmp/./main.py", line 12, in <module>
    with open(os.path.join(tmpdir, f'{i:064}'), 'w') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device: 'tmp/0000000000000000000000000000000000000000000000000000000005590508'

so near 5.6M files.

But also note that the length of files names in the directory matters a lot. If you reduced them down from 64 bytes e.g. with:

    with open(os.path.join(tmpdir, f'{i}'), 'w') as f:

you would then be able to store way more files in the directory, that went up to 20M files and then I lost patience. So you can't just easily put a single specific value to it. I first came across this issue exactly when storing a large number of file with large basenames because the basenames were Bitcoin transaction IDs which have 64 hex bytes each.

Tested on Python 3.11.6, Ubuntu 23.10, Linux kernel 6.5.0 on an ext4 filesystem with:

df -hT

containing:

Filesystem                        Type      Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv ext4      1.9T  1.5T  246G  87% /

and:

sudo file -L -s /dev/mapper/ubuntu--vg-ubuntu--lv ext4

giving:

sudo dumpe2fs /dev/mapper/ubuntu--vg-ubuntu--lv | grep 'Filesystem features'
dumpe2fs 1.47.0 (5-Feb-2023)

giving:

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum

so no large_dir.

Note that the total size of the file names themselves in the above example is 64 B * 10000000 = 640 MB, so there was more than enough space in the disk for them.

All of this just makes part of me just want to use one big indexed SQLite file everywhere. 5 million is a small enough number for a modern computer!

The underlying system call is open itself which can create a file, as opposed to write which writes data to a file, and would instead blow up if you ran out of total space. man 2 open documents:

ERRORS

ENOSPC pathname was to be created but the device containing pathname has no room for the new file.

and we can confirm that the cryptic "28" is ENOSPC by running as per https://unix.stackexchange.com/questions/326766/what-are-the-standard-error-codes-in-linux

errno -ls

which contains:

ENOSPC 28 No space left on device

On the Linux kernel it is at: https://github.com/torvalds/linux/blob/a4145ce1e7bc247fd6f2846e8699473448717b37/include/uapi/asm-generic/errno-base.h#L32

0

I faced a similar issue. The above solutions for deleting the /tmp directory worked for me.

Instead of using the default /tmp location where the service account might not have full access (if following best practices and not using sudo to install Python packages), I moved the /tmp directory to the user's home directory, with TMPDIR environment setting honored by pip install --user ... command.

I faced the issue above of running out space, as mentioned in the answeres above most likely due to so many files/directories being created and not actually running out of volume storage. The solution that worked for me was to delete the /home/<some_domain>/$USER/tmp directory and recreate it every time my continuous deployment pipeline ran. rm -rf /tmp

0

I ran into this problem when running tests. The Django app I was running was inside a docker container, so I had to log in to the container as a root, run rm -r /tmp/ and it fixed the issue.

0

I have the same problem, but found the root cause is "print too frequently" at last. On my code I have some "print(...)" logic which running in loop. When I comment those code, the "No space left on device" error disappeared. Perhaps it is a Python implementation problem. You may try this solution.

For more detail, sample descriptive code will look like following:

loop:
    do some logic 
    call "print" to check result

When your logic code running very fast, you will call "print" very frequently. Then sometimes "[Errno 28] No space left on device" error will appear.

I think this is an implementation limitation of Python "print" code, though I not read the code yet. My Python version is 3.9.7

Thanks for Community reply.

1
  • 3
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented Mar 5, 2022 at 1:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.