47

I have a docker environment with 2 containers (Jenkins and Nexus, both with their own named volume). I have a daily cron-job which deletes unused containers and images. This is working fine. But the problem is inside my devicemapper:

du -sh /var/lib/docker/
30G docker/

I can each folder in my docker folder: Volumes (big, but that's normal in my case):

/var/lib/docker# du -sh volumes/
14G volumes/

Containers:

/var/lib/docker# du -sh containers/
3.2M    containers/

Images:

/var/lib/docker# du -sh image/
5.8M    image/

Devicemapper:

/var/lib/docker# du -sh devicemapper/
  16G   devicemapper/

/var/lib/docker/devicemapper/mnt is 7.3G /var/lib/docker/devicemapper/devicemapper is 8.1G

Docker info:

Storage Driver: devicemapper
 Pool Name: docker-202:1-xxx-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 5.377 GB
 Data Space Total: 107.4 GB
 Data Space Available: 28.8 GB
 Metadata Space Used: 6.148 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.141 GB
 Udev Sync Supported: true

What is this space and am I able to clean this without breaking stuff?

7
  • what rhel and docker version are you on?
    – Matt
    Commented Jun 7, 2016 at 11:59
  • I'm using Ubuntu 14.04 with docker version 1.11.1. Why it using devicemapper which is (rhel/centos) based
    – DenCowboy
    Commented Jun 7, 2016 at 12:09
  • Ubuntu 14.04 doesn't have AUFS in the standard kernel. you need the linux-image-extra kernel. You shouldn't use a loopback devicemapper in any case. I'll do an answer...
    – Matt
    Commented Jun 7, 2016 at 12:20
  • 1
    @Matt thanks for your information. But what for people which are using RHEL's? Is it possible to your devicemapper in a 'good' way?
    – DenCowboy
    Commented Jun 7, 2016 at 12:40
  • Yes, see direct lvm thin pool in the answer
    – Matt
    Commented Jun 7, 2016 at 13:53

4 Answers 4

60

Don't use a devicemapper loop file for anything serious! Docker has big warnings about this. Devicemapper support has also been deprecated from Docker 25.0.

The /var/lib/docker/devicemapper/devicemapper directory contains the sparse loop files that contain all the data that docker mounts. So you would need to use lvm tools to trawl around them and do things. Have a read though the remove issues with devicemapper, they are kinda sorta resolved but maybe not.

I would move away from devicemapper where possible or use LVM thin pools on anything RHEL based. If you can't change storage drivers, the same procedure will at least clear up any allocated sparse space you can't reclaim.

Changing the docker storage driver

Changing storage driver will require dumping your /var/lib/docker directories which contains all your docker data. There are ways to save portions of it but that involves messing around with Docker internals. Better to commit and export any containers or volumes you want to keep and import them after the change. Otherwise you will have a fresh, blank Docker install!

  1. Export data

  2. Stop Docker

  3. Remove /var/lib/docker

  4. Modify your docker startup to use the new storage driver. Set --storage-driver=<name> in /lib/systemd/system/docker.service or /etc/systemd/system/docker.service or /etc/default/docker or /etc/sysconfig/docker

  5. Start Docker

  6. Import Data

OverlayFS

The overlay2 driver is the default preferred storage driver and in use in most places. Just use --storage-driver=overlay2

Direct LVM Thin Pool

Docker removed support for devicemapper in January of 2024 with the release of v25.0. The official documentation currently favors replacing it with overlay2.

Instead of the devicemapper loop file you can use an LVM thin pool directly. RHEL makes this easy with a docker-storage-setup utility that distributed with their EPEL docker package. Docker have detailed steps for setting up the volumes manually.

--storage-driver=devicemapper \
--storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool \
--storage-opt dm.use_deferred_removal=true

Docker 17.06+ supports managing simple direct-lvm block device setups for you.

Just don't run out of space in the LVM volume, ever. You end up with an unresponsive Docker daemon that needs to be killed and then LVM resources that are still in use that are hard to clean up.

9
  • thanks. Great explanation. One little detail to improve your answer. I had to edit /etc/default/docker and add DOCKER_OPTS="--storage-driver=aufs" (probably dependent on your OS)
    – DenCowboy
    Commented Jun 7, 2016 at 15:36
  • I've added that ubu path and a rhel one too. Thanks
    – Matt
    Commented Jun 7, 2016 at 16:07
  • 2
    Just curious what happens when you run out of space / how you can resolve issues arising from that?
    – Josh
    Commented Jun 9, 2016 at 15:55
  • 4
    The Docker daemon becomes unresponsive and stops working. The quickest fix is to delete any volumes and /var/lib/docker and start again. If you have to kill docker you can end up with devicemapper resources remaining open and making everything difficult.
    – Matt
    Commented Jun 9, 2016 at 23:53
  • @Matt In which package is it for CentOS7?
    – lvthillo
    Commented Jul 6, 2016 at 8:20
8

A periodic docker system prune -a works for me on systems where I use devicemapper and not the LVM thinpool. The pattern I use is:

  • I label any containers, images, etc with label "protected" if I want them to be exempt from cleanup
  • I then periodically run docker system prune -a --filter=label!=protected (either manually or on cron with -f)

Labeling examples:

  • docker run --label protected ...
  • docker create --label=protected=true ...
  • For images, Dockerfile's LABEL, eg LABEL protected=true
  • To add a label to an existing image that I cannot easily rebuild, I make a 2 line Dockerfile with the above, build a new image, then switch the new image for the old one (tag).

General Docker label documentation

4

First, what is devicemapper (official documentation)

Device Mapper has been included in the mainline Linux kernel since version 2.6.9 [in 2005]. It is a core part of RHEL family of Linux distributions.

The devicemapper driver stores every image and container on its own virtual device. These devices are thin-provisioned copy-on-write snapshot devices.

Device Mapper technology works at the block level rather than the file level. This means that devicemapper storage driver's thin provisioning and copy-on-write operations work with blocks rather than entire files.

The devicemapper is the default Docker storage driver on some Linux distributions.

Docker hosts running the devicemapper storage driver default to a configuration mode known as loop-lvm. This mode uses sparse files to build the thin pool used by image and container snapshots

Docker 1.10 [from 2016] and later no longer matches image layer IDs with directory names in /var/lib/docker.

However, there are two key directories.

  • The /var/lib/docker/devicemapper/mnt directory contains the mount points for image and container layers.
  • The /var/lib/docker/devicemapper/metadatadirectory contains one file for every image layer and container snapshot.

If your docker info does show your Storage Driver is devicemapper (and not aufs), proceed with caution with those folders.

See for instance issue 18867.

2
  • 1
    There is no clear way to reduce the size of your devicemapper without losing something of my existing containers?
    – DenCowboy
    Commented Jun 7, 2016 at 9:07
  • There are some old bugs regarding trimming sparse files after removes: github.com/docker/docker/issues/3182. It's meant to be fixed but people still report issues.
    – Matt
    Commented Jun 7, 2016 at 13:57
1

I faced the same issue where in my /var/lib/docker/devicemapper/devicemapper/data file has reached ~91% of root volume(~45G of 50G). I tried removing all the unwanted images, deleted volumes, nothing helped in reducing this file.

Did a few googling and understood that the "data" files is loopback-mounted sparse files and docker uses it to store the mount locations and other files we would have stored inside the containers.

Finally I removed all the images which were run before and stopped

Warning: Deletes all docker containers

docker rm $(docker ps -aq)

The reduced the devicemapper file significantly. Hope this may help you .

2
  • 6
    This did not change the size of my devicemapper file. Before: 19G, After: 19G
    – Lee Meador
    Commented May 16, 2019 at 14:58
  • 5
    docker rm $(docker ps -aq) would remove all existing containers. Not sure why one would do that.
    – vinyll
    Commented May 28, 2020 at 6:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.