240

I'm working with a repository with a very large number of files that takes hours to checkout. I'm looking into the possibility of whether Git would work well with this kind of repository now that it supports sparse checkouts but every example that I can find does the following:

git clone <path>
git config core.sparsecheckout true
echo <dir> > .git/info/sparse-checkout
git read-tree -m -u HEAD

The problem with this sequence of commands is the original clone also does a checkout. If you add -n to the original clone command, then the read-tree command results in the following error:

error: Sparse checkout leaves no entry on working directory

How can do the sparse checkout without checking out all the files first?

5

16 Answers 16

176

Please note that this answer does download a complete copy of the data from a repository. The git remote add -f command will clone the whole repository. From the man page of git-remote:

With -f option, git fetch <name> is run immediately after the remote information is set up.


Try this:

mkdir myrepo
cd myrepo
git init
git config core.sparseCheckout true
git remote add -f origin git://...
echo "path/within_repo/to/desired_subdir/*" > .git/info/sparse-checkout
git checkout [branchname] # ex: master

Now you will find that you have a "pruned" checkout with only files from path/within_repo/to/desired_subdir present (and in that path).

Note that on windows command line you must not quote the path, i.e. you must change the 6th command with this one:

echo path/within_repo/to/desired_subdir/* > .git/info/sparse-checkout

if you don't you'll get the quotes in the sparse-checkout file, and it will not work

10
  • 4
    I can't use the command "git checkout [branchname]" (also found error: Sparse checkout leaves no entry on working directory). I've used "git pull origin master" and it works properly.
    – Natty
    Sep 3, 2013 at 7:15
  • 2
    With git version 1.7.2.5 on linux, I got the following results: echo 'dir/*' checks out only the files in dir/ but not in its subdirs; echo 'dir/' (no asterix!) correctly checks out the whole tree under dir/. HTH
    – pavek
    Oct 10, 2013 at 15:01
  • 38
    This just plain didn't work for me - the "git remote" command resulted in the entire repo being checked out - bam! - right then; so the "git config..." and specification of a sub-dir of interest in the following commands had no effect. Is the repo URL specified in the "git remote" command just the path to the top-level .git file? Or should it be a path to the sub-dir of interest? Oct 24, 2013 at 14:44
  • 10
    here's a streamlined version (no need for manually creating the directory, doing an init and remote add, just do the normal git clone+checkout cycle with --no-checkout option as mentioned by @onionjake): git clone --no-checkout <project> cd <project> echo <dir> > .git/info/sparse-checkout git checkout <branch>
    – Gregor
    Aug 10, 2015 at 12:58
  • 23
    The git remote add command downloads everything because that's what -f does -- tells it to immediately fetch, before you've defined the sparse checkout options. But omitting or reordering that isn't going to help. Sparse checkouts affect only the working tree, not the repository. If you want your repository to go on a diet instead, then you need to look at the --depth or --single-branch options instead.
    – Miral
    Dec 11, 2015 at 5:11
96

In 2020 there is a simpler way to deal with sparse-checkout without having to worry about .git files. Here is how I did it:

git clone <URL> --no-checkout <directory>
cd <directory>
git sparse-checkout init --cone # to fetch only root files
git sparse-checkout set apps/my_app libs/my_lib # etc, to list sub-folders to checkout
git checkout # or git switch

Note that it requires git version 2.25 installed. Read more about it here: https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/

UPDATE:

The above git clone command will still clone the repo with its full history, though without checking the files out. If you don't need the full history, you can add --depth parameter to the command, like this:

# create a shallow clone,
# with only 1 (since depth equals 1) latest commit in history
git clone <URL> --no-checkout <directory> --depth 1
11
  • 1
    @mropp, I updated the answer by adding --depth parameter which allows us to do a shallow clone. Will that help? @Tao, not sure how to use --filter in this case, I didn't try it. Could you provide an example, or post another answer to this topic? Jun 3, 2020 at 10:41
  • 7
    note that it doesn't work the same in 2.27 release - I don't know why.
    – Blazes
    Jun 11, 2020 at 16:36
  • 1
    As Blazes said it doesn't work anymore in 2.27, can't find how to make it work again.
    – agemO
    Jun 17, 2020 at 7:27
  • 4
    I think I made that work on 2.28: git clone <url> --no-checkout <dir> cd dir git sparse-checkout set <git dir to checkout> git checkout master This last checkout populates my workdir with the files I needed in <dir to checkout>
    – gxh8Nmate
    Aug 12, 2020 at 3:53
  • 2
    Turns out the behavior in this answer was a bug and doesn't work in git 2.27+. See stackoverflow.com/questions/62423920/… Dec 3, 2020 at 3:00
68

Works in git v2.37.1+

git clone --filter=blob:none --no-checkout --depth 1 --sparse <project-url>
cd <project>

Specify the folders you want to clone

git sparse-checkout add <folder1> <folder2>
git checkout
11
  • 3
    Thank you, looked for an hour for a concise answer, that was up to date, and works. Oct 4, 2021 at 20:06
  • 2
    Is this recursive?
    – tzg
    Feb 3, 2022 at 17:19
  • Works great! Super fast compared to cloning an entire large repo :)
    – Alex
    May 17, 2022 at 21:54
  • Surely you mean 2.37.1…? Jan 31 at 11:10
  • 1
    @DanM. Seems there are two modes: stackoverflow.com/questions/73803120/…
    – NateS
    Apr 21 at 19:08
44

Git clone has an option (--no-checkout or -n) that does what you want.

In your list of commands, just change:

git clone <path>

To this:

git clone --no-checkout <path>

You can then use the sparse checkout as stated in the question.

3
  • 8
    yeah, it doesn't do a checkout, but still does a fetch to download the entire repo history
    – Jason S
    Oct 28, 2015 at 20:40
  • 10
    @JasonS the question was specifically about not doing a checkout. If you do not want then entire history use the --depth <depth> option on git clone. That will only download the last <depth> commits from the history. Currently there is no way to partially download a single commit with git, though if your remote supports it you can use git archive --remote to download partial sets of files.
    – onionjake
    Feb 10, 2016 at 16:53
  • You can now also 'check out' a commit without downloading any files using vfsforgit.org. This might be useful if someone is trying to only checkout a small subset of a single commit.
    – onionjake
    Apr 18, 2019 at 19:10
31

I had a similar use case, except I wanted to checkout only the commit for a tag and prune the directories. Using --depth 1 makes it really sparse and can really speed things up.

mkdir myrepo
cd myrepo
git init
git config core.sparseCheckout true
git remote add origin <url>  # Note: no -f option
echo "path/within_repo/to/subdir/" > .git/info/sparse-checkout
git fetch --depth 1 origin tag <tagname>
git checkout <tagname>
2
  • 3
    --depth 1 is called a shallow clone, just FYI. Jun 1, 2017 at 7:02
  • 1
    Thanks for this. Got it right with this after trying many other ways to prevent downloading the whole repo.
    – J...S
    Oct 17, 2019 at 6:49
12

I found the answer I was looking for from the one-liner posted earlier by pavek (thanks!) so I wanted to provide a complete answer in a single reply that works on Linux (GIT 1.7.1):

1--> mkdir myrepo
2--> cd myrepo
3--> git init
4--> git config core.sparseCheckout true
5--> echo 'path/to/subdir/' > .git/info/sparse-checkout
6--> git remote add -f origin ssh://...
7--> git pull origin master

I changed the order of the commands a bit but that does not seem to have any impact. The key is the presence of the trailing slash "/" at the end of the path in step 5.

5
  • 3
    are you sure this is what you want ? the -f means fetching all the data, you still get all the other information you don't want and it is slow. (This is still " checking out the whole repository")
    – Shuman
    Mar 5, 2016 at 22:45
  • 1
    I tried above steps in Windows but spare checkout does not work in command prompt so I tried Git Bash shell and it worked!!. command prompt is able to execute all the git commands like push, pull etc but when it comes to sparse checkout it fails.
    – user593029
    Apr 6, 2016 at 18:39
  • How to do only files of the subdirectory. I want to only fetch the files inside specific sub directory. Sep 3, 2016 at 4:44
  • @BabishShrestha see comment by onionjake on other answer FWIW :|
    – rogerdpack
    Sep 15, 2016 at 21:47
  • This does not do a sparse clone, not as useful as newer answers that do. BTW, -f forces the full clone. Oct 4, 2021 at 20:09
11

Updated answer 2020:

There is now a command git sparse-checkout, that I present in detail with Git 2.25 (Q1 2020)

nicono's answer illustrates its usage:

git sparse-checkout init --cone # to fetch only root files
git sparse-checkout add apps/my_app
git sparse-checkout add libs/my_lib

It has evolved with Git 2.27 and knows how to "reapply" a sparse checkout, as in here.
Note that with Git 2.28, git status will mention that you are in a sparse-checked-out repository


Note/Warning: Certain sparse-checkout patterns that are valid in non-cone mode led to segfault in cone mode, which has been corrected with Git 2.35 (Q1 2022).

See commit a3eca58, commit 391c3a1, commit a481d43 (16 Dec 2021) by Derrick Stolee (derrickstolee).
(Merged by Junio C Hamano -- gitster -- in commit 09481fe, 10 Jan 2022)

sparse-checkout: refuse to add to bad patterns

Reviewed-by: Elijah Newren
Signed-off-by: Derrick Stolee

When in cone mode sparse-checkout, it is unclear how 'git sparse-checkout'(man) add ... should behave if the existing sparse-checkout file does not match the cone mode patterns.
Change the behavior to fail with an error message about the existing patterns.

Also, all cone mode patterns start with a '/' character, so add that restriction.
This is necessary for our example test 'cone mode: warn on bad pattern', but also requires modifying the example sparse-checkout file we use to test the warnings related to recognizing cone mode patterns.

This error checking would cause a failure further down the test script because of a test that adds non-cone mode patterns without cleaning them up.
Perform that cleanup as part of the test now.


With Git 2.36 (Q2 2022), "git sparse-checkout"(man) wants to work with per-worktree configuration, but did not work well in a worktree attached to a bare repository.

See commit 3ce1138, commit 5325591, commit 7316dc5, commit fe18733, commit 615a84a, commit 5c11c0d (07 Feb 2022) by Derrick Stolee (derrickstolee).
(Merged by Junio C Hamano -- gitster -- in commit 6249ce2, 25 Feb 2022)

worktree: copy sparse-checkout patterns and config on add

Signed-off-by: Derrick Stolee
Reviewed-by: Elijah Newren

When adding a new worktree, it is reasonable to expect that we want to use the current set of sparse-checkout settings for that new worktree.
This is particularly important for repositories where the worktree would become too large to be useful.
This is even more important when using partial clone as well, since we want to avoid downloading the missing blobs for files that should not be written to the new worktree.

The only way to create such a worktree without this intermediate step of expanding the full worktree is to copy the sparse-checkout patterns and config settings during 'git worktree add'(man).
Each worktree has its own sparse-checkout patterns, and the default behavior when the sparse-checkout file is missing is to include all paths at HEAD.
Thus, we need to have patterns from somewhere, they might as well be the current worktree's patterns.
These are then modified independently in the future.

In addition to the sparse-checkout file, copy the worktree config file if worktree config is enabled and the file exists.
This will copy over any important settings to ensure the new worktree behaves the same as the current one.
The only exception we must continue to make is that core.bare and core.worktree should become unset in the worktree's config file.


Original answer: 2016

git 2.9 (June 2016) will generalize the --no-checkout option to git worktree add (the command which allows to works with multiple working trees for one repo)

See commit ef2a0ac (29 Mar 2016) by Ray Zhang (OneRaynyDay).
Helped-by: Eric Sunshine (sunshineco), and Junio C Hamano (gitster).
(Merged by Junio C Hamano -- gitster -- in commit 0d8683c, 13 Apr 2016)

The git worktree man page now includes:

--[no-]checkout:

By default, add checks out <branch>, however, --no-checkout can be used to suppress checkout in order to make customizations, such as configuring sparse-checkout.

10

Sadly none of the above worked for me so I spent very long time trying different combination of sparse-checkout file.

In my case I wanted to skip folders with IntelliJ IDEA configs.

Here is what I did:


Run git clone https://github.com/myaccount/myrepo.git --no-checkout

Run git config core.sparsecheckout true

Created .git\info\sparse-checkout with following content

!.idea/*
!.idea_modules/*
/*

Run 'git checkout --' to get all files.


Critical thing to make it work was to add /* after folder's name.

I have git 1.9

3
  • 3
    Nope, it still downloads everything, all commits and all files, git 2.3.2
    – Tyguy7
    Sep 18, 2015 at 22:00
  • 9
    Sparse checkouts affect only the working tree. They don't affect the repository size or what gets fetched. You need different options if you want that.
    – Miral
    Dec 11, 2015 at 5:13
  • Try Git Bash Shell next time if working in Windows & use above steps by 'pbetkier' it works fine
    – user593029
    Apr 6, 2016 at 18:47
8

Based on this answer by apenwarr and this comment by Miral I came up with the following solution which saved me nearly 94% of disk space when cloning the linux git repository locally while only wanting one Documentation subdirectory:

$ cd linux
$ du -sh .git .
2.1G    .git
894M    .
$ du -sh 
2.9G    .
$ mkdir ../linux-sparse-test
$ cd ../linux-sparse-test
$ git init
Initialized empty Git repository in /…/linux-sparse-test/.git/
$ git config core.sparseCheckout true
$ git remote add origin ../linux
# Parameter "origin master" saves a tiny bit if there are other branches
$ git fetch --depth=1 origin master
remote: Enumerating objects: 65839, done.
remote: Counting objects: 100% (65839/65839), done.
remote: Compressing objects: 100% (61140/61140), done.
remote: Total 65839 (delta 6202), reused 22590 (delta 3703)
Receiving objects: 100% (65839/65839), 173.09 MiB | 10.05 MiB/s, done.
Resolving deltas: 100% (6202/6202), done.
From ../linux
 * branch              master     -> FETCH_HEAD
 * [new branch]        master     -> origin/master
$ echo "Documentation/hid/*" > .git/info/sparse-checkout
$ git checkout master
Branch 'master' set up to track remote branch 'master' from 'origin'.
Already on 'master'
$ ls -l
total 4
drwxr-xr-x 3 abe abe 4096 May  3 14:12 Documentation/
$  du -sh .git .
181M    .git
100K    .
$  du -sh
182M    .

So I got down from 2.9GB to 182MB which is already quiet nice.

I though didn't get this to work with git clone --depth 1 --no-checkout --filter=blob:none file:///…/linux linux-sparse-test (hinted here) as then the missing files were all added as removed files to the index. So if anyone knows the equivalent of git clone --filter=blob:none for git fetch, we can probably save some more megabytes. (Reading the man page of git-rev-list also hints that there is something like --filter=sparse:path=…, but I didn't get that to work either.

(All tried with git 2.20.1 from Debian Buster.)

1
  • 1
    Now the man page of git-rev-list has been modified to reflect the removal of the --filter=sparse:path option: Note that the form --filter=sparse:path=<path> that wants to read from an arbitrary path on the filesystem has been dropped for security reasons.
    – Arnie97
    Oct 27, 2021 at 10:17
8

Yes, Possible to download a folder instead of downloading the whole repository. Even any/last commit

Nice way to do this

D:\Lab>git svn clone https://github.com/Qamar4P/LolAdapter.git/trunk/lol-adapter -r HEAD
  1. -r HEAD will only download last revision, ignore all history.

  2. Note trunk and /specific-folder

Copy and change URL before and after /trunk/. I hope this will help someone. Enjoy :)

Updated on 26 Sep 2019

3
  • 1
    only applicable for those coming from or using svn. Will not upvote this one.
    – C.J.
    Apr 29, 2019 at 21:53
  • @CJohnson as you can see, I'm cloning git repo folder. Working fine
    – Qamar
    Apr 30, 2019 at 5:24
  • 2
    Note that this isn't something git offers out of the box but is something that Github offers adjacent to the regular Git offering. However, it works beautifully when you can utilise it. Thanks! Aug 5, 2019 at 16:42
6

Steps to sparse checkout only specific folder:

1) git clone --no-checkout  <project clone url>  
2) cd <project folder>
3) git config core.sparsecheckout true   [You must do this]
4) echo "<path you want to sparce>/*" > .git/info/sparse-checkout
    [You must enter /* at the end of the path such that it will take all contents of that folder]
5) git checkout <branch name> [Ex: master]
1
  • FYI, in the first(1) step, you no need to use --no-checkout. Just clone the whole repo and then execute all the below steps 2-5 (mentioned above), you will get the output what you want. Let me know if you didn't get it. Jan 5, 2019 at 22:20
5

In git 2.27, it looks like git sparse checkout has evolved. Solution in this answer does not work exactly the same way (compared to git 2.25)

git clone <URL> --no-checkout <directory>
cd <directory>
git sparse-checkout init --cone # to fetch only root files
git sparse-checkout set apps/my_app libs/my_lib # etc, to list sub-folders to checkout
# they are checked out immediately after this command, no need to run git pull

These commands worked better:

git clone --sparse <URL> <directory>
cd <directory>
git sparse-checkout init --cone # to fetch only root files
git sparse-checkout add apps/my_app
git sparse-checkout add libs/my_lib

See also : git-clone --sparse and git-sparse-checkout add

0
4

I'm new to git but it seems that if I do git checkout for each directory then it works. Also, the sparse-checkout file needs to have a trailing slash after every directory as indicated. Someone more experience please confirm that this will work.

Interestingly, if you checkout a directory not in the sparse-checkout file it seems to make no difference. They don't show up in git status and git read-tree -m -u HEAD doesn't cause it to be removed. git reset --hard doesn't cause the directory to be removed either. Anyone more experienced care to comment on what git thinks of directories that are checked out but which are not in the sparse checkout file?

2

In my case, I want to skip the Pods folder when cloning the project. I did step by step like below and it works for me. Hope it helps.

mkdir my_folder
cd my_folder
git init
git remote add origin -f <URL>
git config core.sparseCheckout true 
echo '!Pods/*\n/*' > .git/info/sparse-checkout
git pull origin master

Memo, If you want to skip more folders, just add more line in sparse-checkout file.

2

I took this from TypeScript definitions library @types:

Let's say the repo has this structure:

types/
|_ identity/
|_ etc...

Your goal: Checkout identity/ folder ONLY. With all its contents including subfolders.

⚠️ This requires minimum git version 2.27.0, which is likely newer than the default on most machines. More complicated procedures are available in older versions, but not covered by this guide.

git clone --sparse --filter=blob:none --depth=1 <source-repo-url>
git sparse-checkout add types/identity types/identity ...

This will check out the types/identity folder to your local machine.

--sparse initializes the sparse-checkout file so the working directory starts with only the files in the root of the repository.

--filter=blob:none will exclude files, fetching them only as needed.

--depth=1 will further improve clone speed by truncating commit history, but it may cause issues as summarized here.

0

Accepted answer not fully worked as needed for me, because it also downloads files from root folder.

This is how I downloaded only one folder that I needed:

git clone --no-checkout --depth=1 --filter=blob:none --branch=$(VERSION) $(REPO_URL) && \
cd $(REPO_NAME) && \
git config core.sparseCheckout true && \
echo "$(FOLDER)/" >> .git/info/sparse-checkout && \
git checkout

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.