3

in my project, i had by mistake added some big image files to my repo. i read up on GitHub how to remove files from the history, and it did work: you cannot see the files in the history anymore. BUT then i made a tar.gz from my project for backup, and it is now twice the size it used to have! i haven't added anything else that could justify this increase, so my suspicion is that the repo data that used to represent the image files was not really thrown out of the repo. can someone corroborate that? is there a fix?

edit to clarify i know pretty little about git so i took exactly the steps as indicated on the GitHub help pages, with the single exception that i had to use a force switch from the second file onwards, as in git filter-branch -f --index-filter ....

to partially answer my own question, i think i could create a second git repo without the unwanted materials by

  • creating an empty repo in a different location
  • reproducing the file situation at different steps of my project, leaving out unwanted ones
  • and finally use that new repo instead of the old to push materials to GitHub.

has that been done before? specifically, can i use that new git repo instead of the old one with the same project on GitHub?

btw, for what it's worth, this is about a presentation i am writing right now; there is an image of the tower of Babel in it that existed in several versions in high resolution, which explains the size of the problem (~100MB of unwanted data).

edit 2 thx a lot for suggestions; i did

rm -rf .git/refs/original/
git reflog expire expire=now --all
git reflog expire --all
git gc --aggressive --prune=now

with the effect that the *.tar.gz size got smaller by a mere 0.5%...

edit 3 it is daunting to experience the sheer complexity that is git. i'm giving up at this point. i did a test with a small throw-away repo; i did an initial commit, added a big file, did a commit, removed the file and tried to erase its traces from memory with

rm very-big-file.xcf
git filter-branch --index-filter 'git rm --cached --ignore-unmatch very-big-file.xcf' --prune-empty -- --all
rm -rf .git/refs/original/
git reflog expire --all
git gc --aggressive --prune=now

these are the recorded *.tar.gz sizes:

foo.tar.gz          7,518 
foo2.tar.gz    65,735,003 
foo3.tar.gz    32,777,155 

the big file's compressed size is 32,955,246 bytes, which makes it entirely plausible that it is still fully present under .git, maybe even in uncompressed form.

GIT Y U SO STUBBORN??

isn't there any git purge extension to do this? i mean, git filter-branch --index-filter 'git rm --cached --ignore-unmatch very-big-file.xcf' --prune-empty -- --all is not exactly what i could type from memory when i have a slight hangover.

4
  • 1
    Can you possibly add a little more about how you removed them? Did you run git gc? Did you remove it from the index with git rm --cached?
    – Nic
    Mar 24, 2012 at 20:21
  • @CaptainGiraffe that isn't necessary in this case.
    – Nic
    Mar 24, 2012 at 20:22
  • 1
    You followed the git filter-branch instructions and did it on all branches, right? Did you then do the rm -rf .git/refs/original/ and git reflog expire=now --all and git gc --prune=now steps?
    – torek
    Mar 24, 2012 at 21:02
  • Oops, have a typo in my reflog expire above. Anwyay let me expand on your third update...
    – torek
    Mar 25, 2012 at 19:43

2 Answers 2

0

A quick way is to get the history to look exactly like you want, add the repo as the remote of a new empty one and then just fetch. You will only get the references and objects in the history they represent.

You can now push this to a new GitHub repo.

0

Re "edit 3"... here's a complete sequence, which I actually logged and retried to eliminate typos this time. :-) Note that you can't filter-branch after removing the big file unless you commit that remove (which is kind of pointless for this example). Check the du -s output.

$ git init bigoop
Initialized empty Git repository in /tmp/bigoop/.git/
$ cd bigoop
$ echo tiny file with not much in it > tiny
$ git add tiny
$ git commit -m 'initial commit'
[master (root-commit) bd07e5a] initial commit
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 tiny
$ cp /path/to/huge/file hugefile
$ git add hugefile
$ git commit -m 'oops, add huge file'
[master 25cd764] oops, add giant file
 1 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 hugefile
$ du -s .git
618992  .git
$ rm hugefile
$ git filter-branch --index-filter 'git rm --cached --ignore-unmatch hugefile' --prune-empty -- --all
Cannot rewrite branch(es) with a dirty working directory.
$ git checkout hugefile
$ git filter-branch --index-filter 'git rm --cached --ignore-unmatch hugefile' --prune-empty -- --all
Rewrite 25cd7647f49173fa8f42c0ca0a2ab8baf1842fca (2/2)rm 'hugefile'

Ref 'refs/heads/master' was rewritten
$ du -s .git
619012  .git
$ rm -rf .git/refs/original/
$ git reflog expire --expire=now --all
$ git gc --prune=now
Counting objects: 3, done.
Writing objects: 100% (3/3), done.
Total 3 (delta 0), reused 0 (delta 0)
$ du -s .git
140     .git

As for "GIT Y U SO STUBBORN??" ... it really works hard not to lose stuff. Even when you're trying to make it lose stuff. :-)

1
  • ok so that looks like i missed out something; will retry later today
    – flow
    Mar 26, 2012 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.