126

I have some csv files that are larger than github's file size limit of 100.00 MB. I have been trying to use the Git Large File Storage extension.

https://git-lfs.github.com/

From LFS - "Large file versioning- Version large files—even those as large as a couple GB in size—with Git."

I have applied the following on the folders of concern:

git lfs track "*.csv"

However, when I push:

remote: error: File Time-Delay-ftn/Raw-count-data-minor-roads1.csv is 445.93 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File Time-Delay-ftn/Raw-count-data-major-roads.csv is 295.42 MB; this exceeds GitHub's file size limit of 100.00 MB

When I look at the folder in question:

-rw-r-----   1 user  staff    42B 23 Oct 12:34 .gitattributes
-rw-r--r--   1 user  staff   1.3K 19 Oct 14:32 DfT_raw_major_manipulation.py
-rw-r--r--   1 user  staff   1.2K 16 Oct 15:08 DfT_raw_minor_manipulation.py
drwxr-xr-x  21 user  staff   714B 22 Oct 11:35 Driving/
-rwxr-xr-x@  1 user  staff   295M 19 Oct 14:47 Raw-count-data-major-roads1.csv*
-rwxr-xr-x@  1 user  staff   446M 16 Oct 14:52 Raw-count-data-minor-roads1.csv*

when I vim the .gitattributes file you can see the lfs setup:

*.csv filter=lfs diff=lfs merge=lfs -text

What am I doing wrong?

UPDATE

When I query

git lfs ls-files

I get nothing returned. This indicates that despite the .csv filter being successfully applied to the .gitattributes file the csv files are not being picked up by lfs

7
  • “You will not be allowed to push files larger than 100 MB.” – GitHub.com on Working with large files
    – Gumbo
    Commented Oct 25, 2015 at 14:45
  • 3
    Could you not zip them or something instead to decrease size?
    – Henry
    Commented Oct 25, 2015 at 15:29
  • 4
    @Gumbo That document also says "consider introducing Git Large File Storage (Git LFS)". This appears to be what is asked about. Following that link the docs say "It's the ideal solution for pushing files to GitHub that are larger than 100 MB" Commented Oct 26, 2015 at 20:47
  • @natb1 - yes, if you read the documentation - that is exactly what it says. Commented Oct 27, 2015 at 7:49
  • @Henry. Thanks - I tried that. Still too large in some cases Commented Oct 27, 2015 at 7:49

13 Answers 13

188

Since Git LFS 2.2.0 you can use git lfs migrate command. In your case it would look this way:

git lfs migrate import --include="*.csv"

After this, you should be able to push changes into origin.

More information about git lfs migrate is available on Git LFS 2.2.0 release note site and git-lfs-migrate command documentation.

8
  • 2
    Worked Like a charm after trying all the above answers, issue caused by Zoom. Commented Mar 4, 2021 at 13:05
  • 12
    This is a more modern answer, should be bumped to the top. For anyone reading this, try this before any of the above answers. Commented Mar 18, 2021 at 6:06
  • 4
    If you get the not a valid git command error, make sure to install lfs. A how to guide for all platforms can be found here. - stackoverflow.com/a/48734334/2480714 Commented Apr 24, 2021 at 4:16
  • Can anyone explain --include flag for me? and what is the difference between --include and --exclude? Thanks in advance.
    – gia huy
    Commented Oct 12, 2021 at 3:56
  • Can I specify remote in this command? Because I have two remotes in my repo and I can only access one of them at this time(The LAN one is not accessible).
    – K.F
    Commented May 7, 2022 at 1:52
44

i had same issue, but it's resolved using filter-branch

git filter-branch --tree-filter 'rm -rf path/to/your/file' HEAD

it'll take some time if you have big project, then push it

git push

1
  • 1
    Only one that works. git was still failing to push due to some deleted file that was too large.
    – NelsonGon
    Commented Sep 7, 2022 at 21:55
36

Simply adding git-lfs configuration to an existing repository will not retroactively convert your large files to LFS support. Those large files will remain in your history and GitHub will refuse your pushes.

You need to rewrite your history to introduce git-lfs to your existing commits. I recommend the BFG repo cleaner tool, which added LFS support recently.

You should be able to convert historical usage of your CSV files by:

$ java -jar ~/bfg-1.12.5.jar --convert-to-git-lfs '*.csv' --no-blob-protection
4
  • 1
    thanks! indeed. However, I am not having much luck with bfg. I did brew install bfg. I have checked the file path but I am getting Error: Unable to access jarfile /usr/local/cellar/bfg-1.12.5.jar Commented Nov 10, 2015 at 16:57
  • 1
    @LearningSlowly - I'm not familiar with brew. I would think that you would only need to download the jar from GitHub and run it per the instructions. Commented Nov 11, 2015 at 17:43
  • I have a further issue - but this definitely solved the problem I had here! Many thanks Commented Nov 12, 2015 at 12:43
  • Tried the bfs approach for about 3 hours. It doesn't seem to do anything: Found 133 commits Cleaning commits: 100% (133/133) Cleaning commits completed in 216 ms. BFG aborting: No refs to update - no dirty commits found?? Then makes anti trump comments, which I don't care about. I do care that its unintuitive and didn't work
    – AlexK
    Commented Dec 7, 2019 at 0:42
27

I have faced the same problem while some of my csv file was merged in previous commit and github rejected my push. Later I found this command which worked for me.

git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch fixtures/11_user_answer.json'
# for my case
git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch tensorflow.csv'

filter-branch rewrites commit history of the mentioned branches.

--index-filter is filter for rewriting the index

git rm --cached --ignore-unmatch tensorflow.csv completely forgets the tensorflow.csv file from all commits it is present.

Original post link

documentation for filter-branch

4
  • 1
    How do I chain multiple files? Its complaining I have several that I need to remove from history
    – Potion
    Commented Oct 5, 2020 at 23:00
  • what is the extension type? If all are csv, then try *.csv. Or the specific extension type you want to remove.
    – avijit
    Commented Oct 6, 2020 at 4:49
  • this deleted my big file locally
    – avr-girl
    Commented May 14, 2021 at 12:09
  • " With git rm --cached you stage a file for removal, but you don't remove it from the working dir. The file will then be shown as untracked. " for details stackoverflow.com/a/38001958/4374376
    – avijit
    Commented Nov 1, 2022 at 21:14
25

Try the following command in your terminal. It will resolve the lfs issue.

git lfs migrate import --include="*.csv"
3
  • it works. somehow the git-lfs.com does not mention it lol
    – Alpha
    Commented May 30, 2023 at 20:14
  • @Alpha It does mention it! > Note that defining the file types Git LFS should track will not, by itself, convert any pre-existing files to Git LFS, such as files on other branches or in your prior commit history. To do that, use the git lfs migrate(1) command, which has a range of options designed to suit various potential use cases. Commented May 24 at 9:32
  • I did that, it doesnt list the issuing file anymore (with migrate info), but gitlab still complains about the >100MB blobs, I have 5 unpushed commits, and it lists 3 blobs of >100MB, its only one file tho. It was my understanding that migrate import would fix this for any unpushed commit Commented Aug 14 at 7:05
20

I had this error:

remote: error: File client/static/static-version/20171221_221446.psd is 223.61 MB; this exceeds GitHub's file size limit of 100.00 MB

And because I already removed this file from this folder, created .gitignore file and tried to commit couple times, I didn't know that it was cached, I could not push to github. In my case helped:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch client/static/static-version/20171221_221446.psd'

Where I placed full file path(from error above) to remove it from cache. After that, push was made successfully

1
  • 1
    Thank you! this solved my issues but in the git help I couldn't find any mention about this
    – Omiod
    Commented Dec 2, 2021 at 16:34
17

I hit the same problem yesterday and cracked it. I was unable to push, and it appeared that none of my big files were in lfs.

There is probably a better way, but this worked for me. I have a large repo with 2.5 gigs of data.

I setup a new repo then setup lfs in it. git lfs init

I then configured my various file types git lfs track "*.pdb" git lfs track "*.dll" I then commmited my changes and pushed.

I then added my big files. I used sourcetree, and in the output notes it would state for the big files matching my wildcards that it was committing tiny txt file instead. (sorry, I didn't record these, but it should be obvious).

Then I pushed, and I saw 'skipping files', and the push succeeded quickly.

so the problem is probably trying to add files to lfs that are already in your history. You can only add new files. You can probably clean your repo of these files.

Note: I did find that quite a few files that matched my wildcards were not picked up by lfs. Similar files in different folders were picked up, but not all. I tried explicitly adding these files using the full path. git lfs track "Windows/bin/myBigFile.dll" but that didn't help either. In the end I gave up due to time constraints.

You should also check your storage limit with gitHub. I purchased the extra 50gig to cover my requirements.

Cloning the repo now downloads the files separately and everything is finally working well.

2
  • Thanks! A viable solution. However, I think using the BFG tool is the best way forward. I ended up in the same boat though, had to purchase 50GB ;) Commented Nov 16, 2015 at 15:33
  • The important part here is to update your .gitattributes (git track <>) in a separate commit from adding the large files. Commented May 19, 2019 at 5:30
13

This solution worked for me on Ubuntu 20.04

Install the Git Large File Storage (git-lfs)

sudo apt-get install git-lfs
git-lfs install

To convert any pre-existing files to Git LFS, such as files on other branches or in your prior commit history us the git lfs migrate command

git lfs migrate import --include="*.ipynb" 

Select the file types to be tracked

git lfs track "*.ipynb"

update the gitattributes

git add .gitattributes

Now push to the git git commit -m "commit message" git push

And for the storage and bandwidth usage please refer

https://docs.github.com/en/github/managing-large-files/versioning-large-files/about-storage-and-bandwidth-usage

1
  • thanks. that actually worked very smoothly.
    – Joschi
    Commented Sep 15, 2021 at 7:34
6

If you know which commit introduced the large commit, you could also try squashing that commit with the subsequent commits that introduced Git LFS.

For example, if the large commit was three commits ago (as revealed by git status), you could do the following:

git rebase -i HEAD~3

Then, replace all "pick" usages after the first one with "squash" in the interactive dialog.

Then,

git push origin --force
0

I also, faced this error.I installed LFS did the following steps

enter image description here

And this is what I did:

Step 1: Get the commit ID suppose "3540ea5c94ccd63d40" (commit ID that contain large file)

Step 2: git rebase -i HEAD~1

Step 3: Replace all "pick" to "squash" in the interactive dialog.

Step 4: Press esc and then save file by :wq.

enter image descrip
tion here

Step 5: the run git push origin BRANCH_NAME

And it's solved

enter image description here

0

If you need totally remove file from GIT history use this command:

git filter-repo --path path-to-file-from-current-dir --invert-paths

--invert-paths - used for delete exactly --path you need else it will remove all but that file

filter-repo - need to be installed

more details here https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository

-1

This may be help you enter image description here

Click-OriginalWebPage

Only install lfs to a exist repo may be not enough. Your may also change the commit history. Hope this is work for you .

-2

It looks like you haven't initialised git-lfs. Try to type

git lfs init

Source: Installing Git LFS

3
  • 1
    Try to make a new commit and push again
    – hombit
    Commented Oct 28, 2015 at 5:22
  • 1
    Thanks. Still struggling. When i run git lfs ls-files it comes back blank Commented Nov 10, 2015 at 14:36
  • This is not the correct Answer@LearningSlowly See this one confluence.atlassian.com/bitbucket/…
    – linjiejun
    Commented Aug 8, 2019 at 3:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.