38

I need to make the repo smaller. I think I can make it smaller by removing problematic binary files from git history:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch BigFile'

And then releasing the objects:

rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --aggressive --prune=now

(Feel free to comment if those commands are wrong.)

The problem: How to identify those big files so that I can asses whether to remove them from git history? Most likely they are not in the working tree anymore - they have been deleted and probably also untracked with:

git rm --cached BigFile

5 Answers 5

31

twalberg's answer does the trick. I wrapped it up in a loop so that you can list files in order by size:

while read -r largefile; do
    echo $largefile | awk '{printf "%s %s ", $1, $3 ; system("git rev-list --all --objects | grep " $1 " | cut -d \" \" -f 2-")}'
done <<< "$(git rev-list --all --objects | awk '{print $1}' | git cat-file --batch-check | sort -k3nr | head -n 20)"

head -n 20 restricts the output to the top 20. Change as necessary.

Once you've identified the problem files, check out this answer for how to remove them.

1
24

You can find the hash IDs of the largest objects like this:

git rev-list --all --objects | awk '{print $1}' | git cat-file --batch-check | sort -k3nr

Then, for a particular SHA, you can do this to get the file name:

git rev-list --all --objects | grep <SHA>

Not sure if there's a more efficient way to do it. If you know for sure that everything is in pack files (not loose objects), git verify-pack -v produces output that includes the size, and I seem to remember seeing a script somewhere that would parse that output and match each object back up with the original files.

13

Couldn't help optimizing MatrixManAtYrService's answer:

git rev-list --all --objects | git cat-file --batch-check='%(objectname) %(objecttype) %(objectsize) %(rest)' | grep blob | sort -k3nr | head -n 20

This way git rev-list is called only once (and not per object being displayed), and the script is more clear.

3
  • I haven't been able to figure out how to use numfmt go get a human readable size, but this converts it to mb: git rev-list --all --objects | git cat-file --batch-check='%(objectname) %(objecttype) %(objectsize) %(rest)' | sort -k3nr | awk '$3=$3 / 1024 / 1024' | head -n 20
    – GammaGames
    Commented Jan 24, 2020 at 15:59
  • thank you, this is much faster than the original answer
    – Recep
    Commented Aug 13, 2020 at 19:44
  • 1
    This command only gives the size of the biggest blob. Is there a way to display the summarized size of all versions of the file ? because e.g. when I file of 10MB only is change once it takes less space than a 2MB file that is changed 200 times in the history
    – Radon8472
    Commented Oct 19, 2020 at 14:41
3

Using git-filter-repo you can analyze space taken up in the repository.

git filter-repo --analyze

To see the largest deleted files, see:

.git/filter-repo/analysis/path-deleted-sizes.txt
1

I wrote a script that will tell you the largest objects, files, or directories in my answer here. Without arguments, it'll tell you the size of all objects, sorted by size. You can tell it --sum or --directories to sum all the objects for each file and print that, or to do the same for all files in each directory. I hope it's useful!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.