99

We updated our private docker registry to the official Registry 2.0. This version can now delete docker images identified by a hashtag (see https://docs.docker.com/registry/spec/api/#deleting-an-image) but I still don't see a way to cleanup old images.

As our CI server is continously producing new images, I would need a method to delete all images from the private registry which are no longer identified by a named tag.

If there's no built-in way to achieve this, I think a custom script could possibly work, but I don't see a v2 API method either to list all stored hashtags of an image..

How can I keep my private registry clean? Any hints?

3
  • Actually, the situation is worse than you think. The spec is not yet implemented; calling DELETE returns a 400. See github.com/docker/distribution/issues/422 Jun 10, 2015 at 5:01
  • private registry has a long way to go
    – xdays
    Jul 12, 2015 at 7:32
  • 7
    Hello from the future (4 years later) - is there a good method to deal with this nowadays? The options seem abut the same as they did when this question was active.
    – emmdee
    Jul 26, 2019 at 4:22

9 Answers 9

70

Deletion of images (you can keep 10 last versions, like I do in my CI) is done in three steps:

  1. Enable image deletion by setting environment variable REGISTRY_STORAGE_DELETE_ENABLED: "true" and passing it to docker-registry

  2. Run below script (it will delete all images and tags but keep last 10 versions)

    registry.py -l user:pass -r https://example.com:5000 --delete --num 10

  3. Run garbage collection (you can put it into your daily cron task)

    docker-compose -f [path_to_your_docker_compose_file] run registry bin/registry garbage-collect /etc/docker/registry/config.yml

registry.py can be downloaded from the link below, it also allows listing images, tags and layers, as well as deleting a particular image and/or tag.

https://github.com/andrey-pohilko/registry-cli

Before garbage collection my registry folder was 7 Gb, after I ran the above steps it deflated down to 1 Gb.

5
  • The thing with this solution is that it can't deal with... untagged revisions (?). Those you push to a registry without specifying a tag. @anoxis Can you confirm? I was able to free 120 Gb by deleting those. But there were issues, I'm about to add a comment to that answer.
    – x-yuri
    Jan 16, 2019 at 13:47
  • @x-yuri probably you're right. My script is not intended to clean up all revisions, like in the solution you used. Known issue with the script is that it cannot delete the entire image from registry. But, as you can see, the original question is about how to clean old tags from CI server, not all the revisions or all repos. So, both solutions are good, they just serve different purposes.
    – anoxis
    Jan 17, 2019 at 16:09
  • I don't want to delete all revisions (that would basically delete a tag), not to say all repositories. It's your script that basically deletes all revisions of tags by deleting tags themselves. And I'm in no way trying to devalue your answer. I'm just adding details regarding my case. From what I can see the registry.py uses the api (/v2/_catalog for repositories,
    – x-yuri
    Jan 18, 2019 at 0:25
  • .../v2/REPO/tags/list for tags). But the things is, every tag generally has more than one revision. One can see those at /var/lib/docker/volumes/registry/_data/docker/registry/v2/repositories/REPO/_manifests/tags/TAG/index. And those can be deleted using DELETE /v2/REPO/manifests/sha256:HASH.
    – x-yuri
    Jan 18, 2019 at 0:25
  • ...But the filesystem doesn't seem to accurately reflect the state of affairs, and that might get you by deleting tags, not just old revisions of a tag (you delete all revisions but the last, but the last one exists only on disk). At least, that's the best explanation of what went wrong that I've got so far. Hopefully, the official solution does what I think it does.
    – x-yuri
    Jan 18, 2019 at 0:27
63

Regarding your question:

I would need a method to delete all images from the private registry which are no longer identified by a named tag

A new version of the docker registry in distribution/registry:master has this nice feature! However, you won't be able to trigger it from the API.

Anyway, you will be able to clean all untagged manifests in your registry, meaning that every overwritten tag won't leave old manifests and blobs in the registry. Every "unused" layer will be cleaned by the Registry Garbage Collectior.

You just have to run a docker exec:

docker exec ${container_id} registry garbage-collect \ 
  /path/to/your/registry/config.yml \
  --delete-untagged=true

Looking at this garbage-collect binary help:

Usage: 
  registry garbage-collect <config> [flags]
Flags:
  -m, --delete-untagged=false: delete manifests that are not currently referenced via tag
  -d, --dry-run=false: do everything except remove the blobs
  -h, --help=false: help for garbage-collect

You can have a look at the github PR. It's been merged and usable with distribution/registry, tag master as of 2018-02-23. It supersedes the docker/docker-registry project with a new API design, focused around security and performance...

I did use this feature today and recovered 89% of registry space (5.7 GB vs. 55 GB). Then I switched back to stable registry.

6
  • 4
    The changes from the PR are supposed to be in the official image now.
    – x-yuri
    Jan 16, 2019 at 14:34
  • 3
    Right now this should be the accepted answer. It does exactly what was required. If someone is running a private repo, the only thing required is to just pull the new image docker pull registry:2, stop the container docker-compose stop registry, remove it docker-compose rm registry and recreate it docker-compose up -d registry. The above will work if you use docker-compose and execute in the directory with docker-compose.yml AND your container is names registry :)
    – Krystian
    Mar 25, 2019 at 13:34
  • The garbage collector is now in Docker Registry v2.7+, so any freshly installed Docker Registry should have it. Nov 15, 2019 at 1:15
  • 2
    This is a good answer. However I had no idea where or what my "registry file" was. I found its location by doing a docker exec -it <docker-registry-container-name> /bin/sh Then performed a find / -iname "*.yml" Which uncovered the location of the file.
    – Dave
    Apr 18, 2021 at 18:02
  • Can this be done automatically somehow? Aug 31, 2023 at 21:05
33

I host regestry in docker container with name docker-registry_registry_1 from image: registry:2

I just run garbage-collect with -m

docker exec docker-registry_registry_1 bin/registry garbage-collect /etc/docker/registry/config.yml -m
0
21

This is doable, although ugly. You need to be running (I think) registry 2.3 or greater, and have enabled deleting (REGISTRY_STORAGE_DELETE_ENABLED=True env var or equivalent). The example commands below assume a local filestore in /srv/docker-registry, but I'd be surprised if something equivalent couldn't be cooked up for other storage backends.

For each repository you wish to tidy up, you need to enumerate the digest references that are no longer required. The easiest way to do this is per-tag, using latest as an example in this case, you'd do something like:

ls -1tr /srv/docker-registry/v2/repositories/<repo>/_manifests/tags/latest/index/sha256 \
| tail -n +3

This will list all but the three most recent digests pushed to the latest tag. Alternately, if you don't care about tags so much, but just want to keep the last few references pushed, you can do:

ls -1t /srv/docker-registry/v2/repositories/<repo>/_manifests/revisions/sha256 \
| tail -n +3

Then, you just delete the references you don't need:

for hash in $(ls -1t /srv/docker-registry/v2/repositories/<repo>/_manifests/tags/latest/index/sha256 | tail -n +3); do
  curl -X DELETE https://registry:5000/v2/<repo>/manifests/sha256:$hash
done

Finally, you need to do a GC run, because the registry implements "soft deletes", which doesn't actually delete anything, it just makes it unavailable:

docker exec docker-registry /bin/registry \
  garbage-collect /path/to/config.yml

Yes, this is all messy as hell, grovelling around in the backend storage, because there's no API method to enumerate all digests associated with a given tag, but that's the way the cookie crumbles.

5
  • Minor clarification: the flag is REGISTRY_STORAGE_DELETE_ENABLED, not ..._STORE_.... See docs.docker.com/registry/configuration/#/… for more details. Aug 18, 2016 at 14:54
  • 3
    In v2.6 the directory structure is .../_manifest/tags/<tag>/index/.... Mar 7, 2017 at 12:27
  • @womble It somehow deleted a tag (latest) of one of my images. Also, state of the filesystem doesn't seem to reflect the state of affairs. After deleting unneeded manifests, and running garbage collector, I can still see at least some manifests that are supposed to be gone at /srv/docker-registry/v2/repositories/<repo>/_manifests/tags/latest/index/sha256.
    – x-yuri
    Jan 16, 2019 at 13:51
  • 1
    It doesn't surprise me that things have changed in the time since I wrote that answer. I no longer use the Docker registry for anything I don't absolutely have to.
    – womble
    Jan 17, 2019 at 22:57
  • I'm not sure they have changed. I might just be lucky enough to run into the case where it doesn't work properly. My conjecture is that filesystem doesn't fully reflect state of affairs. And when you delete all but the last revision, the latter might be only on disk. So you've just deleted all the revisions, and the tag disappears as well. Hopefully, the official solution does what I think it does. What do you use by the way?
    – x-yuri
    Jan 18, 2019 at 0:33
7

There is some discussion happening to design this - right now, there is no layer cleanup tool / endpoint.

I would encourage you to go to:

and/or reach out on Freenode IRC on #docker-distribution for more.

7
  • 1
    so assuming that we burn over 50Gb disk space a day in our CI server, what's your suggested solution to the problem? That IRC channel also looks pretty quiet to me :) Apr 23, 2015 at 13:01
  • Short terms, here are suggestions: if that's "throwable" content, you can wipe out your entire storage once in a while. If that's not acceptable, 50GB a day makes for 5T in three months - maybe some cheap storage solution can work out? Also, I would expect a real solution coming in the next couple of months. Sorry for not having a better solution right away... Apr 23, 2015 at 17:48
  • @MangledDeutz any news on this? it's been several months now.. thanks!
    – yorch
    Jan 26, 2016 at 17:57
  • @yorch There is now a pending PR to officially support GC here: github.com/docker/distribution/pull/1386 - and there are community maintained solutions as well. Cheers. Jan 26, 2016 at 18:48
  • @MangledDeutz great news! thanks, I hope it can be merged back soon
    – yorch
    Jan 26, 2016 at 20:46
4

I pieced together various parts of this thread and created an easy to use cleanup script in bash You can check it out in this gist cleanup.sh

2
  • 1
    Thanks for your script. This worked for me with some modifications: 1) The for hash in loop assumes that repo names are a single string, while mine are named project/repo, so I had to add another inner loop to navigate the extra folder level. 2) Added authentication to the curl command with -u usr:pwd. 3) Added REGISTRY_STORAGE_DELETE_ENABLED=true to the env vars for running the registry.
    – foz
    Mar 16, 2018 at 15:36
  • 1
    @foz It's better to use -n, --netrc option in place of -u.
    – x-yuri
    Jan 15, 2019 at 21:54
3

I was looking for the same functionality in the registry v2 api but only found soft deleting which is not what I was looking for. While researching I found the Github project delete-docker-registry-image which removes the actual files from the mounted volume via a bash script. Not tested it maybe useful...

1
  • it seems to be doing the trick (in my environment :) I had to create a symlink 'ln -s /var/lib/docker/volumes/c49189f29d8bd93f644438dee774685790687a67c576eb1349cbfe218e14fc20/_data /opt/registry_data' as I run my registry inside a container and the cryptic string is the mount of the container (to be retrieved by the command docker inspect --format '{{range .Mounts}}{{.Name}}{{end}}' <containername>) Jul 15, 2016 at 7:56
2

For removing unsed images, three steps manually on these sequence:

  1. docker rmi -f **imageid**

  2. rm -Rf /home/**homedirectory**/docker-registry/data/docker/registry/v2/repositories/**yoursystemname**/**yourimagename**/_manifests/tags/**image version**/

  3. docker exec $(docker ps -q) bin/registry garbage-collect /etc/docker/registry/config.yml -m

*Pay attention:

** You must execute those commands (above) in test environment, because if you commit any mistake or didn't understand any step, you don't damage your production environment.

** You can schedule those commands (above) using crontab as root. In the step 3) you must execute removing "-it", as result: docker exec $(docker ps -q) bin/registry garbage-collect /etc/docker/registry/config.yml -m`.

It works for me for more than 6 months.

0

Kubernetes equivalent:

kubectl exec --tty --stdin registry-0 -n devops -- registry garbage-collect /etc/docker/registry/config.yml --delete-untagged=true

1
  • that would be quite particular to your setup Jan 16 at 2:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.