367

I got some docker containers running on AWS EC2, the /var/lib/docker/overlay2 folder grows very fast in disk size.

I'm wondering if it is safe to delete its content? or if docker has some kind of command to free up some disk usage.


UPDATE:

I actually tried docker system prune -a already, which reclaimed 0Kb.

Also my /docker/overlay2 disk size is much larger than the output from docker system df

After reading docker documentation and BMitch's answer, I believe it is a stupid idea to touch this folder and I will try other ways to reclaim my disk space.

5
  • 5
    did you find any answer to this? I am still getting the same issue. Jun 1, 2020 at 13:28
  • 22
    I ran docker image prune --all and then docker system prune -a. It has reclaimed my disk space by around 50 GB, which was being used up by files under /var/lib/docker/overlay2. But, docker system prune -a would have been enough. Also, my configuration specifics are : OS: Ubuntu 20, Docker : 19.03.12 Sep 21, 2020 at 10:23
  • 1
    Prune didn't help me instead try this docker volume rm $(docker volume ls -qf dangling=true More here: stackoverflow.com/a/75928154/1753177
    – lambodar
    Apr 4 at 10:16
  • 3
    If you are uncertain whether docker system prune -a really catches everything, you could run the following find to identify and folders ignored by docker. find /var/lib/docker/overlay2 -mindepth 1 -maxdepth 1 -type d \! -exec bash -c "docker ps -q | xargs docker inspect --format '{{.GraphDriver.Data}}'| grep -q "{} \; -print Note this will identify any folder that is NOT mentioned in the GraphDirver by any of the running docker images. You should find a folder called l which contains a list of links to all the other folders. Jun 27 at 13:12
  • Following the comment from @TobiOetiker I took the risk to manually delete the output of the find command (stoping Docker completely first), except for the 'l' forlder. So far it restarted and runs ok, and I recovered ~30GB of disk space.
    – Simon
    Nov 27 at 11:13

23 Answers 23

332

Docker uses /var/lib/docker to store your images, containers, and local named volumes. Deleting this can result in data loss and possibly stop the engine from running. The overlay2 subdirectory specifically contains the various filesystem layers for images and containers.

To cleanup unused containers and images, see docker system prune. There are also options to remove volumes and even tagged images, but they aren't enabled by default due to the possibility of data loss:

$ docker system prune --help

Usage:  docker system prune [OPTIONS]

Remove unused data

Options:
  -a, --all             Remove all unused images not just dangling ones
      --filter filter   Provide filter values (e.g. 'label=<key>=<value>')
  -f, --force           Do not prompt for confirmation
      --volumes         Prune volumes

What a prune will never delete includes:

  • running containers (list them with docker ps)
  • logs on those containers (see this post for details on limiting the size of logs)
  • filesystem changes made by those containers (visible with docker diff)

Additionally, anything created outside of the normal docker folders may not be seen by docker during this garbage collection. This could be from some other app writing to this directory, or a previous configuration of the docker engine (e.g. switching from AUFS to overlay2, or possibly after enabling user namespaces).

What would happen if this advice is ignored and you deleted a single folder like overlay2 out from this filesystem? The container filesystems are assembled from a collection of filesystem layers, and the overlay2 folder is where docker is performing some of these mounts (you'll see them in the output of mount when a container is running). Deleting some of these when they are in use would delete chunks of the filesystem out from a running container, and likely break the ability to start a new container from an impacted image. See this question for one of many possible results.


To completely refresh docker to a clean state, you can delete the entire directory, not just sub-directories like overlay2:

# danger, read the entire text around this code before running
# you will lose data
sudo -s
systemctl stop docker
rm -rf /var/lib/docker
systemctl start docker
exit

The engine will restart in a completely empty state, which means you will lose all:

  • images
  • containers
  • named volumes
  • user created networks
  • swarm state
10
  • 3
    The overlay2 folder should contain the filesystem layers needed for your images and containers. You're free to ignore this advice, but please don't ask me for advice on how to recover a failed system after you break it, particularly since I gave you a supported way to cleanup your filesystem.
    – BMitch
    Oct 10, 2017 at 23:18
  • 71
    I tried docker system prune -a, which recovered 0kb space. Right now the case for me is that /docker/overlay2 disk size is much larger than the output from docker system df. That's the reason why I keep digging this issue. Again, thanks for your reply sir. I guess I need to read more about docker documentation or probably erase the docker entirely and restart it. I only have a postgres database need to keep, and I did mount it
    – qichao_he
    Oct 10, 2017 at 23:23
  • 4
    docker system prune -a -f worked for me on Docker version 17.09.0-ce but only after stopping all my containers which is a real bummer. Had anyone luck with a newer version?
    – Bruno
    Oct 28, 2018 at 18:11
  • 24
    I will say that the "supported" way isn't working for me neither. Doing all the docker system prune -a, docker volume prune, docker image prune and docker container prune still leaves me with 80% of my disk being used by Docker. That's with all containers stopped. Jan 7, 2019 at 10:52
  • 7
    I get that this is the "official" answer, but it's quite simply buggy as hell and leaves behind huge amounts of data, particularly overlay2 diff folders for images and containers that are no longer around. My docker should be using about 100G, but it's using over 200G.
    – Novaterata
    May 7, 2021 at 12:56
131

I found this worked best for me:

docker image prune --all

By default Docker will not remove named images, even if they are unused. This command will remove unused images.

Note each layer in an image is a folder inside the /usr/lib/docker/overlay2/ folder.

7
  • 4
    'image prune' worked much better than 'system prune'. Thanks!
    – DavidG
    Apr 15, 2020 at 13:24
  • 10
    Warning! This is pretty desctructive, as it removes all images of non-running containers. You will be re-building them for hours if they are yours and not yet pushed to the registry. But it still cannot go beyond what docker system df shows (you may be still out of space and that evil overlay2 garbage dump needs to be nuked manually.
    – mirekphd
    Apr 18, 2020 at 21:35
  • 4
    Well, yeah, it removes the images.
    – Sarke
    Apr 20, 2020 at 3:16
  • 1
    Attention: In my case where I used docker swarm it also deleted all tagged images, even for running containers
    – velop
    Oct 18, 2020 at 20:17
  • Is there a command to find which overlay belongs to container? Oct 9, 2022 at 14:52
58

I had this issue... It was the log that was huge. Logs are here :

/var/lib/docker/containers/<container id>/<container id>-json.log

You can manage this in the run command line or in the compose file. See there : Configure logging drivers

I personally added these 3 lines to my docker-compose.yml file :

my_container:
  logging:
    options:
      max-size: 10m
4
  • 2
    Can you add a few lines from the link to the answer? Apr 9, 2019 at 7:49
  • Would be nice to also get info on how to identify which container is the one with the giant log file. I have a bunch of containers and log files, some are huge some are tiny. Apr 18, 2019 at 5:13
  • How's that answers the OP question?! Oct 20, 2019 at 11:39
  • 3
    This answer is a partial answer, especially if 'logs' were the problem (maybe we can improve it with some edits?). Until I saw this answer, I was about to start randomly deleting big directories from my overly full overlay2. In my case, the total capacity for /var/lib/docker was 50GB and 36GB of it was consumed by one file: /var/lib/docker/overlay2/<container id>/diff/var/log/faillog. On the assumption that this file isn't central to keeping everything running, my short term hack is to just remove it (and maybe I'll also adjust my docker-compose too).
    – D. Woods
    Nov 12, 2019 at 21:55
41

also had problems with rapidly growing overlay2

/var/lib/docker/overlay2 - is a folder where docker store writable layers for your container. docker system prune -a - may work only if container is stopped and removed.

in my i was able to figure out what consumes space by going into overlay2 and investigating.

that folder contains other hash named folders. each of those has several folders including diff folder.

diff folder - contains actual difference written by a container with exact folder structure as your container (at least it was in my case - ubuntu 18...)

so i've used du -hsc /var/lib/docker/overlay2/LONGHASHHHHHHH/diff/tmp to figure out that /tmp inside of my container is the folder which gets polluted.

so as a workaround i've used -v /tmp/container-data/tmp:/tmp parameter for docker run command to map inner /tmp folder to host and setup a cron on host to cleanup that folder.

cron task was simple:

  • sudo nano /etc/crontab
  • */30 * * * * root rm -rf /tmp/container-data/tmp/*
  • save and exit

NOTE: overlay2 is system docker folder, and they may change it structure anytime. Everything above is based on what i saw in there. Had to go in docker folder structure only because system was completely out of space and even wouldn't allow me to ssh into docker container.

3
  • Thank for this answer, we put in containers an old data-base/app that generate a lot of /var/log/apache2/error.log. I reset error.log and access.log and add a new volume to allow easiest management
    – bcag2
    Oct 12, 2020 at 12:15
  • 4
    Just a small nitpick: you should edit the crontab using the crontab -e command, which checks validity before saving. Apr 13, 2021 at 1:31
  • Your answer was very useful for my investigations on this problem. Thanks.
    – EM90
    May 12, 2021 at 9:40
27

If your system is also used for building images you might have a look at cleaning up garbage created by the builders using:

docker buildx prune --all

and

docker builder prune --all
5
  • 3
    This answers deserves way more attention. Instead of having to clean up the entire Docker system, this saved me 64GB while keeping all the images intact. Much appreciated.
    – Martin
    Nov 2, 2022 at 16:43
  • Very useful, thanks! This made the biggest difference for me.
    – hedleyroos
    Jan 18 at 7:40
  • 3
    Thanks, this command removed 360.7GB of docker cache on my build server. I was only removing images, containers and volumes and forgot about build cache.
    – Crayon
    May 31 at 14:57
  • This should be the accepted answer. Made a big difference.
    – bradbajuz
    Nov 13 at 15:58
  • docker system prune -a should clear the build cache (according to its warnings) but doesn't! Nov 21 at 1:06
19

Backgroud

The blame for the issue can be split between our misconfiguration of container volumes, and a problem with docker leaking (failing to release) temporary data written to these volumes. We should be mapping (either to host folders or other persistent storage claims) all of out container's temporary / logs / scratch folders where our apps write frequently and/or heavily. Docker does not take responsibility for the cleanup of all automatically created so-called EmptyDirs located by default in /var/lib/docker/overlay2/*/diff/*. Contents of these "non-persistent" folders should be purged automatically by docker after container is stopped, but apparently are not (they may be even impossible to purge from the host side if the container is still running - and it can be running for months at a time).

Workaround

A workaround requires careful manual cleanup, and while already described elsewhere, you still may find some hints from my case study, which I tried to make as instructive and generalizable as possible.

So what happened is the culprit app (in my case clair-scanner) managed to write over a few months hundreds of gigs of data to the /diff/tmp subfolder of docker's overlay2

du -sch /var/lib/docker/overlay2/<long random folder name seen as bloated in df -haT>/diff/tmp

271G total

So as all those subfolders in /diff/tmp were pretty self-explanatory (all were of the form clair-scanner-* and had obsolete creation dates), I stopped the associated container (docker stop clair) and carefully removed these obsolete subfolders from diff/tmp, starting prudently with a single (oldest) one, and testing the impact on docker engine (which did require restart [systemctl restart docker] to reclaim disk space):

rm -rf $(ls -at /var/lib/docker/overlay2/<long random folder name seen as bloated in df -haT>/diff/tmp | grep clair-scanner | tail -1)

I reclaimed hundreds of gigs of disk space without the need to re-install docker or purge its entire folders. All running containers did have to be stopped at one point, because docker daemon restart was required to reclaim disk space, so make sure first your failover containers are running correctly on an/other node/s). I wish though that the docker prune command could cover the obsolete /diff/tmp (or even /diff/*) data as well (via yet another switch).

It's a 3-year-old issue now, you can read its rich and colorful history on Docker forums, where a variant aimed at application logs of the above solution was proposed in 2019 and seems to have worked in several setups: https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604

1
  • 5
    thank you for actually looking into the real problem
    – Novaterata
    May 7, 2021 at 13:00
14

Friends, to keep everything clean you can use de commands:

docker system prune -a && docker volume prune
2
  • This is work for me.I had a not referenced docker volumes.I'm executed "docker system prune" command and it's removed unused docker volumes. Thanks @Tiago Barreto
    – waruna k
    Jul 2, 2021 at 10:59
  • 9
    BE CAREFUL docker volume prunewill remove all your data that was persisted from docker to host disk........ system prune is safe though
    – Herz3h
    Jul 23, 2021 at 9:39
12

WARNING: DO NOT USE IN A PRODUCTION SYSTEM

/# df
...
/dev/xvda1      51467016 39384516   9886300  80% /
...

Ok, let's first try system prune

#/ docker system prune --volumes
...
/# df
...
/dev/xvda1      51467016 38613596  10657220  79% /
...

Not so great, seems like it cleaned up a few megabytes. Let's go crazy now:

/# sudo su
/# service docker stop
/# cd /var/lib/docker
/var/lib/docker# rm -rf *
/# service docker start
/var/lib/docker# df
...
/dev/xvda1      51467016 8086924  41183892  17% /
...

Nice! Just remember that this is NOT recommended in anything but a throw-away server. At this point Docker's internal database won't be able to find any of these overlays and it may cause unintended consequences.

4
  • 2
    Completely hosing the /var/lib/docker directory (while the daemon is stopped and assuming the directory contains no special filesystem mounts or similar) in fact is a valid quick-and-dirty way to get back to square one. I'm not sure why you're getting all the downvotes. Docker tries to be self-healing, and it will recognize when all hope is lost and reinitialize the /var/lib/docker directory as needed.
    – L0j1k
    Sep 9, 2019 at 19:52
  • 2
    Holy **** finally a working answer. I've been pruning and doing stuff for 4 hours but i should have just stopped the docker service, put everything in the trash and the restarting it.
    – Osi
    Oct 9, 2020 at 18:04
  • It works, but it also just deletes EVERYTHING that Docker has produced. So not a good solution, per sé.
    – Akito
    Nov 23, 2020 at 11:52
  • I have a building node that picks up jobs from Jenkins to build images, upload them, then delete them. It kept piling up as builds were going through it. I do docker system prune --all --force before every build, but that still didn't help. The overlay2 directory was taking over 70GB. I added these steps to my build process, and it's now running without issues. Jun 13 at 15:42
7

"Official" answer, cleaning with "prune" commands, does not clean actually garbage in overlay2 folder.

So, to answer the original question, what can be done is:

Disclaimer: Be careful when applying this. This may result broking your Docker object!

  • List folder names (hashes) in overlay2
  • Inspect your Docker objects (images, containers, ...) that you need (A stopped container or an image currently not inside any container do not mean that you do not need them).
  • When you inspect, you will see that it gives you the hashes that are related with your object, including overlay2's folders.
  • Do grep against overlay2's folders
  • Note all folders that are found with grep
  • Now you can delete folders of overlay2 that are not referred by any Docker object that you need.

Example:

Let say there are these folders inside your overlay2 directory,

a1b28095041cc0a5ded909a20fed6dbfbcc08e1968fa265bc6f3abcc835378b5
021500fad32558a613122070616963c6644c6a57b2e1ed61cb6c32787a86f048

And what you only have is one image with ID c777cf06a6e3.

Then, do this:

docker inspect c777cf06a6e3 | grep a1b2809
docker inspect c777cf06a6e3 | grep 021500

Imagine that first command found something whereas the second nothing.

Then, you can delete 0215... folder of overlay2:

rm -r 021500fad32558a613122070616963c6644c6a57b2e1ed61cb6c32787a86f048

To answer the title of question:

  • Yes, it is safe deleting dxirectly overlay2 folder if you find out that it is not in use.
  • No, it is not safe deleting it directly if you find out that it is in use or you are not sure.
3
  • It's not clear that you verified the layer was not being used, you'd need to check every container and image since layers are shared between images. If the containers are stopped and images are unused, a docker prune -a will remove the images and layers that aren't used by any images.
    – BMitch
    Jan 3, 2022 at 19:38
  • 1
    Yes, I have checked every of them. You can write commands to check all at once if you have thousands of them. As I have already stated in answer, prune does not work for garbage! Somehow, after deleting every possible little Docker object with prune, overlay2 was fulled with some GBs of garbage! Docker's management of disk is far away from being intelligent. Moreover, prune is very dangerour command! My container might be stopped but it does not necessarily mean that I do not need it! It even does not warn, which images, containers etc it will delete if I confirm the command. Jan 3, 2022 at 19:41
  • from the /var/lib/docker/overlay2 directory you can use this micro script to answer the question quickly: for dir in `\ls`; do echo $dir; docker inspect ad8cf8bd0bdb | grep $dir | wc; done. You'll have to replace ad8cf8bd0bdb with your own CONTAINER ID. Any which returns a count of 0 is a candidate for cleanup. Aug 26 at 16:27
5

adding to above comment, in which people are suggesting to prune system like clear dangling volumes, images, exit containers etc., Sometime your app become culprit, it generated too much logs in a small time and if you using an empty directory volume (local volumes) this fill the /var partitions. In that case I found below command very interesting to figure out, what is consuming space on my /var partition disk.

du -ahx /var/lib | sort -rh | head -n 30

This command will list top 30, which is consuming most space on a single disk. Means if you are using external storage with your containers, it consumes a lot of time to run du command. This command will not count mount volumes. And is much faster. You will get the exact directories/files which are consuming space. Then you can go to those directories and check which files are useful or not. if these files are required then you can move them to some persistent storage by making change in app to use persistent storage for that location or change location of that files. And for rest you can clear them.

3

Based on Mert Mertce's answer I wrote the following script complete with spinners and progress bars.

Since writing the script, however, I noticed the extra directories on our build servers to be transient - that is Docker appears to be cleaning up, albeit slowly. I don't know if Docker will get upset if there is contention for removing directories. Our current solution is to use docuum with a lot of extra overhead (150+GB).

#!/bin/bash
[[ $(id -u) -eq 0 ]] || exec sudo /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")"
progname=$(basename $0)
quiet=false
no_dry_run=false
while getopts ":qn" opt
do
    case "$opt" in
      q)
          quiet=true
          ;;
      n)
          no_dry_run=true
          ;;
      ?)
          echo "unexpected option ${opt}"
          echo "usage: ${progname} [-q|--quiet]"
          echo "    -q: no output"
          echo "    -n: no dry run (will remove unused directories)"
          exit 1
          ;;
    esac
done
shift "$(($OPTIND -1))"

[[ ${quiet} = false ]] || exec /bin/bash -c "$(printf '%q ' "$BASH_SOURCE" "$@")" > /dev/null

echo "Running as: $(id -un)"

progress_bar() {
    local w=80 p=$1;  shift
    # create a string of spaces, then change them to dots
    printf -v dots "%*s" "$(( $p*$w/100 ))" ""; dots=${dots// /.};
    # print those dots on a fixed-width space plus the percentage etc.
    printf "\r\e[K|%-*s| %3d %% %s" "$w" "$dots" "$p" "$*";
}

cd /var/lib/docker/overlay2
echo cleaning in ${PWD}
i=1
spi=1
sp="/-\|"
directories=( $(find . -mindepth 1 -maxdepth 1 -type d | cut -d/ -f2) )
images=( $(docker image ls --all --format "{{.ID}}") )
total=$((${#directories[@]} * ${#images[@]}))
used=()
for d in "${directories[@]}"
do
    for id in ${images[@]}
    do
        ((++i))
        progress_bar "$(( ${i} * 100 / ${total}))" "scanning for used directories ${sp:spi++%${#sp}:1} "
        docker inspect $id | grep -q $d
        if [ $? ]
        then
            used+=("$d")
            i=$(( $i + $(( ${#images[@]} - $(( $i % ${#images[@]} )) )) ))
            break
        fi
    done
done
echo -e "\b\b " # get rid of spinner
i=1
used=($(printf '%s\n' "${used[@]}" | sort -u))
unused=( $(find . -mindepth 1 -maxdepth 1 -type d | cut -d/ -f2) )
for d in "${used[@]}"
do
    ((++i))
    progress_bar "$(( ${i} * 100 / ${#used[@]}))" "scanning for unused directories ${sp:spi++%${#sp}:1} "
    for uni in "${!unused[@]}"
    do
        if [[ ${unused[uni]} = $d ]]
        then
            unset 'unused[uni]'
            break;
        fi
    done
done
echo -e "\b\b " # get rid of spinner
if [ ${#unused[@]} -gt 0 ]
then
    [[ ${no_dry_run} = true ]] || echo "Could remove:  (to automatically remove, use the -n, "'"'"no-dry-run"'"'" flag)"
    for d in "${unused[@]}"
    do
        if [[ ${no_dry_run} = true ]]
        then
            echo "Removing $(realpath ${d})"
            rm -rf ${d}
        else
            echo " $(realpath ${d})"
        fi
    done
    echo Done
else
    echo "All directories are used, nothing to clean up."
fi
3

I navigated to the folder containing overlay2. Using du -shc overlay2/*, I found that there was 25G of junk in overlay2. Running docker system prune -af said Total Reclaimed Space: 1.687MB, so I thought it had failed to clean it up. However, I then ran du -shc overlay2/* again only to see that overlay2 had only 80K in it, so it did work.

Be careful, docker lies :).

3

For me prune with image and volume didn't work.

First check the disk space uses. You can relay on ncdu; the best disk space utility cli I can across. It will display directory wise space occupied and there are many useful build-in control to manage filesystem. This will give you a fair idea which particular process and directory occupies more memory.

sudo ncdu -x /var 

Coming back to docker, once you are sure that docker is the one which takes more disk space. You can try prune it and if prune is not cleaning try to clear dangling volume using below command. This will not delete any container or any volume in use.

docker volume rm $(docker volume ls -qf dangling=true)

Also as a standard practice limit container logs. By default, Docker will store container logs indefinitely. You can limit the amount of disk space used by container logs by setting a limit in the Docker daemon configuration file (/etc/docker/daemon.json). For example, you could add the following line to limit container logs to 50MB. If there is not file present with daemon.json you can add one.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m"
  }
}

After making changes to the daemon configuration file, you'll need to restart the Docker daemon with the sudo service docker restart command.

2

DON'T DO THIS IN PRODUCTION

The answer given by @ravi-luthra technically works but it has some issues!

In my case, I was just trying to recover disk space. The lib/docker/overlay folder was taking 30GB of space and I only run a few containers regularly. Looks like docker has some issue with data leakage and some of the temporary data are not cleared when the container stops.

So I went ahead and deleted all the contents of lib/docker/overlay folder. After that, My docker instance became un-useable. When I tried to run or build any container, It gave me this error:

failed to create rwlayer: symlink ../04578d9f8e428b693174c6eb9a80111c907724cc22129761ce14a4c8cb4f1d7c/diff /var/lib/docker/overlay2/l/C3F33OLORAASNIYB3ZDATH2HJ7: no such file or directory

Then with some trial and error, I solved this issue by running

(WARNING: This will delete all your data inside docker volumes)

docker system prune --volumes -a

So It is not recommended to do such dirty clean ups unless you completely understand how the system works.

2

In my case, systemctl stop docker then systemctl start docker somehow automatically free space /var/lib/docker/*

1

I had the same problem, in my instance it was because ´var/lib/docker´ directory was mounted to a running container (in my case google/cadvisor) therefore it blocked docker prune from cleaning the folder. Stopping the container, running docker prune -and then rerunning the container solved the problem.

0

Everything in /var/lib/docker are filesystems of containers. If you stop all your containers and prune them, you should end up with the folder being empty. You probably don't really want that, so don't go randomly deleting stuff in there. Do not delete things in /var/lib/docker directly. You may get away with it sometimes, but it's inadvisable for so many reasons.

Do this instead:

sudo bash
cd /var/lib/docker
find . -type f | xargs du -b  | sort -n

What you will see is the largest files shown at the bottom. If you want, figure out what containers those files are in, enter those containers with docker exec -ti containername -- /bin/sh and delete some files.

You can also put docker system prune -a -f on a daily/weekly cron job as long as you aren't leaving stopped containers and volumes around that you care about. It's better to figure out the reasons why it's growing, and correct them at the container level.

0

Docker apparently keeps image layers of old versions of an image for running containers. It may happen if you update your running container's image (same tag) without stopping it, for example:

docker-compose pull
docker-compose up -d

Running docker-compose down before updating solved it, the downtime is not an issue in my case.

0

I recently had a similar issue, overlay2 grew bigger and bigger, But I couldn’t figure out what consumed the bulk of the space.

df showed me that overlay2 was about 24GB in size.

With du I tried to figure out what occupied the space… and failed.

The difference came from the fact that deleted files (mostly log files in my case) where still being used by a process (Docker). Thus the file doesn’t show up with du but the space it occupies will show with df.

A reboot of the host machine helped. Restarting the docker container would probably have helped already… This article on linuxquestions.org helped me to figure that out.

0

Maybe this folder is not your problem, don't use the result of df -h with docker. Use the command below to see the size of each of your folders:

echo; pwd; echo; ls -AlhF; echo; du -h --max-depth=1; echo; du-sh
0

The reason why the overlay2 directory is full is because one of the containers has log records. At least so was my docker. Docker stop and start solves this but it's a temporary solution. You will either close the logs or the containers will be restarted every day.

You can see the areas covered by the containers with the following command.

du -sh /var/lib/docker/containers/*
0

In my case, my disk was filling up and I found that the /tmp directory in my webserver container was full of old PNG files. Apparently, these are not being unlinked by the application once uploaded.

My solution was far simpler than everything here. I just went into the container:

docker exec -it {container_name} bash

and deleted /tmp/*.png and I released a ton of disk space. Sometimes the solution is simple and it's almost always a tmp directory filling up.

-2
docker system prune -af && docker image prune -af
2
  • 3
    This is very dangerous. Use without the f option. For me it saved my images, this can delete images if they are currently not running. Use this command instead: docker system prune -a && docker volume prune
    – Eran Yogev
    Dec 16, 2021 at 10:43
  • 1
    Another note: the -f means force, -a means all and will still delete images if you continue. The difference is it will prompt you with a warning if you don't add the "f"
    – Eran Yogev
    Mar 13, 2022 at 13:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.