4

I am trying to add a folder that has to be tracked, however it is > 1GB / > 20k files.

As soon I add it, I will get warning about too many active changes and Git simply will refuse to work correctly (on VS Code after the warning I cannot commit anything is it says there is no changes (yet left side button shows that there are changes)).

Tried to use command line but simply doesn't push new folder :/

Any advice how to deal with lots of new files ? I will have ~ 5-6x folders between 1GB - 5GB and from 20k - 200k filer per folder.

7
  • 1
    You could try to add a subset of the files, commit, add another subset, amend with git commit --amend, and so on. This will still create (in the end) 1 commit with all the files, you just slowly build up the commit. Not sure it will matter to whichever tool is complaining though, but it might be worth a try. Commented May 22, 2019 at 13:21
  • Unfortunately cannot do it properly as each folder has hundreds of folders with different number of files. Does it mean that Git won't work correctly with higher number of files ? Do you know perhaps what is the limit for git ?
    – Mike
    Commented May 22, 2019 at 13:22
  • 1
    It may just be VS Code that don't want to show in a visual control all those files, have you tried solely using the command line? Commented May 22, 2019 at 13:26
  • Perfect., worked with git add . and git commit However it happens also with Git Extensions. Thanks!
    – Mike
    Commented May 22, 2019 at 13:30
  • I'd think any visual tool will have some limits. To be honest I don't think it's all that typical to add that many files in one commit, so you may just have tripped a rare usecase that those tools aren't built to handle. Commented May 22, 2019 at 13:31

2 Answers 2

2

I've ran into similar problems with VS Code. If you are currently stuck where VS Code git extension keep repeating this message, put the offending folder in .gitignore then delete the git object cache for it.

git rm -r --cached path/to/folder

A good strategy for large amount of files in VS Code is to compartmentalize each folder and commit separately as stated by Lasse.

Something like in .gitignore structure of splitting the folder like this

folder/folderA
folder/folderB
folder/folderC

Remove one at a time and commit, and be as granular as required. Rinse and repeat. And if you have phpstorm use that instead for these large commits.

0

Should you really be pushing this many files to a git repo? Adding multiple gigabytes of files to a git repo is considered a bad practice.

For example Github.com requests that git repos be less than 1GB

Consider the following approaches

Git LFS

https://git-lfs.github.com/
Git LFS is a plugin that is designed to store binary files. They are managed with git, but are stored externally. This is considered the best approach for when you must have large files in a git repo.

Dependency Resolver

If the files that are being added to the git repo are external dependencies, consider looking at a dependency resolver to fetch the dependencies on demand.

Javascript/Node - npm
Ruby - bundler

1
  • 1
    OP's project isn't large, it's just too large for VS Code.
    – jthill
    Commented May 23, 2019 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.