261

I am trying to git clone the LibreOffice codebase, but at the moment I have an internet connection of about 300kbps and it's just anything but stable. I can get the connection back any moment, but then the git clone process already stopped working, and no way to get it running again. Is there some way to have a more failure-resistant git clone download?

One option I considered myself is to download someone else's .git directory, but that is overly dependent of others and doesn't seem like the best possible solution to me.

6

18 Answers 18

176

Two solutions (or rather workarounds) that come to mind are:

  • Use shallow clone i.e. git clone --depth=1, then deepen this clone using git fetch --depth=N, with increasing N. You can use git fetch --unshallow (since 1.8.0.3) to download all remaining revisions.

  • Ask somebody to bundle up to some tagged release (see git-bundle(1) manpage). The bundle itself is an ordinary file, which you can download any way, via HTTP/FTP with resume support, via BitTorrent, via rsync, etc. Then you can create clone from bundle, fix configuration, and do further fetches from official LibreOffice repository.

9
  • 3
    The shallow clone trick doesn't work well in practice. Cloning a well-packed repo (git://libvirt.org/libvirt.git) changes a 68M transfer into a 61M + 35M transfer. A feature to prioritise the worktree, rather than all branches at depth 1, might fare better; session resumption would be better still.
    – Tobu
    Jan 19, 2012 at 12:09
  • 2
    @Tobu: Shallow clone trick might work in repository with lonG history. There is ongoing work to make shallow clone get only a single branch by default. That might have helped. Or not. Jan 19, 2012 at 15:53
  • 9
    This works really well now, with git 1.7.10. The initial depth=1 clone of the Git repository is only 4.72Mb, while the whole repository is 55Mb. Further fetches can be as small as you want, (depth=100 gave me a ~20Mb fetche). The total compressed download was 31Mb, over one clone and 3 fetches.
    – naught101
    Mar 26, 2013 at 9:01
  • 2
    @naught101 It downloads objects for one revision, and if source code itself is large (not history), then it will be an issue again...
    – kan
    Mar 28, 2013 at 13:06
  • 8
    for m in $(seq 1 50);do git fetch --depth=$[m*100];done worked for me, thanks! :)
    – Trass3r
    Jan 15, 2019 at 11:41
80

I don't think this is ready yet. There's an old GSoC page that which planned to implement your desired feature. My best bet is, like you suggested download it as a directory. I'm assuming you are able to resume downloads over other protocols.

Restartable Clone

When cloning a large repository (such as KDE, Open Office, Linux kernel) there is currently no way to restart an interrupted clone. It may take considerable time for a user on the end of a small pipe to download the data, and if the clone is interrupted in the middle the user currently needs to start over from the beginning and try again. For some users this may make it impossible to clone a large repository.

Goal: Allow git-clone to automatically resume a previously failed download over the native git:// protocol. Language: C Mentor: Shawn Pearce Suggested by: Shawn Pearce on gmane


Update

Along with the shallow cloning (git clone --depth=1) suggestion in one of the other answers it may be helpful if someone can make a bare repository for you if you can communicate with the provider. You can easily convert the bare repository to a full repository. Also read the comments in that answer as a shallow clone may not always help.

12
  • Thanks for the information, so my problem is known and a solution is worked on... What would you recommend as a work-around?
    – LaPingvino
    Oct 17, 2010 at 19:31
  • 16
    Well just yesterday ,I Lost my 600 rupees($10) Because of this Problem.Internet Bandwidth is quite precious thing in my Part of the world. Dec 24, 2013 at 14:06
  • 2
    Lots of people asking for updates and nobody sharing their contribution to the solution. Apr 16, 2017 at 0:45
  • 2
    Mar'18 - lukin for it still...on this earth!!
    – earthling
    Mar 23, 2018 at 5:11
  • 4
    11 years later, Google's attack on the underlying socioeconomic issue of unreliable bandwidth with Google Fiber and Google Fi had mixed results. Its fiber micro-trenches in the city of Louisville were cut too shallowly into the asphalt, and the cables were found popping out from the road surface soon after work. Meanwhile, --depth 1 and --unshallow appears to have withstood the years of usage.
    – rwong
    Feb 8, 2019 at 22:39
18

This method uses 3rd party server.

First, do git clone --bare, then rsync -v -P -e ssh user@host:repo.git . You can use msys under Windows.

2
  • I tried --bare option, it created the expected contents of .git internal files inside repo.git , I had to do the git clone file:///path/to/repo.git/ to get the actual repository
    – PiyusG
    May 25, 2016 at 9:00
  • 2
    Linus doesn't own GitHub…by "3rd-party server", did you actually mean “Git server which does not jail its users so heavily as to prohibit their use of rsync(1) by the way GitHub I'm looking at you”? Or, do you mean to first git clone on a 3rd-party server and then rsync it to the local machine? Jul 17, 2018 at 18:11
17

I would like to put my 5 cents here. This is actually what helped me to solve this issue

  • Turn off compression
  • Increase http.postBuffer
  • Do a partial clone
  • Navigate to the cloned directory and fetch the rest of the clone
  • Pull the rest
git config --global core.compression 0
git config --global https.postBuffer 524288000
git clone  <your_git_http_url_here> --depth 1
git fetch --unshallow 
git pull --all

This helped me to clone ~3GB repo over the 8Mbps adsl connection, of course I had to perform fetch and pulls few times, but still ...

16

"Never underestimate the bandwidth of a carrier pigeon and a bundle of SD cards" would be the modern form of this answer. Tar it up, plain old cp -a it, whatever, and mail the damn thing. Find someone willing to take two minutes of their time to drop a thumb drive into an SASE. Find a contact, there, they might even do it for you.

0
13

Increasing buffer size will help you in this problem. Just follow the steps.

  1. Open terminal or Git Bash and with cd go to the location where you wanted to clone repo.

  2. Set compression to 0

    git config --global core.compression 0
    
  3. Set postBuffer size

    git config --global http.postBuffer 1048576000
    
  4. Set maxRequestBuffer size

    git config --global http.maxRequestBuffer 100M
    
  5. Now start clone

    git clone <repo url>
    
  6. Wait till clone completes.

1
  • 2
    This should definitely be the accepted answer. It solves the problem. Jul 2, 2020 at 7:17
11

You can "download someone else's .git directory", but with that someone else being the official repository itself. The LibreOffice repositories are available via http, for instance their build.git is at http://anongit.freedesktop.org/git/libreoffice/build.git/ (see http://cgit.freedesktop.org/libreoffice/ for the complete list, the http URL is at the bottom of each repository's page).

What you see at these http URLs is nothing more than a .git directory (actually a "bare" repository, which has only what you would find in the .git directory). It is the same directory the server for the git:// protocol (git daemon) would read. If you make a copy of these directories with a web downloader (for instance wget -m -np), you can clone from your copy and it will work as well as if you had cloned directly from the http repository.

So, what you can do is: for each repository, get a copy of it with your favorite web downloader (which will deal with all the issues with resuming broken downloads), and clone from that copy. When you want to update, use again your favorite web downloader to update your copy, and pull from that copy. Now your clones and updates are as resistant to bad connections as your favorite web downloader is.

3
  • They made the conversion to just one repository now, trying your tip wget decides to download the site at once however... (trying again now, will probably update here later...)
    – LaPingvino
    Aug 8, 2011 at 13:36
  • Your command seems to get all links on the site, which is not what is meant to happen. I resorted to write a script that seems to work here: gist.github.com/1307703 Anyway, thanks a lot for the initial idea!
    – LaPingvino
    Oct 23, 2011 at 18:48
  • Interesting idea, I'm trying to get the ruby/ruby repo from github and I'm getting blocked by the robots.txt... any suggestions?
    – hanetzer
    Dec 2, 2014 at 3:47
10

Let's break git clone down into its component parts, and use git checkout to prevent re-downloading files.

When git clone runs, the first few things it does are equivalent to

git init
git remote add origin <repo_url>
git fetch origin <branch>

If you run the above steps manually, and assuming that they completed correctly, you can now run the following as many times as necessary:

git checkout --force <branch>

Note that it will checkout all files each time it's run, but you will not have to re-download files, which may save you a ton of time.

5
  • 2
    it doesn't work the way you describe, it will not allow to do a git reset after a broken fetch
    – MaikoID
    Nov 8, 2017 at 13:07
  • As I said, once you assume that a fetch has completed successfully, you can run git reset. If your fetch is broken, then reset won't work. You need to either A) repeatedly try to fetch again until it works, or B) abandon this and try something else.
    – cowlinator
    Nov 8, 2017 at 23:01
  • I did something else I it miraculous worked. I did a git pull instead of git fetch =)
    – MaikoID
    Nov 11, 2017 at 1:35
  • 1
    @MaikoID I believe a git pull is just calling git fetch internally, and then merges, so the command should ned have made the difference
    – lucidbrot
    Sep 23, 2018 at 10:16
  • Fetch still restarts from the beggining if it fails. it just create a new tmp file at .git/objects/pack. I saw you said fetch should complete correctly, but it doesnt differ from clone command in the end, at least to download huge projects like unreal engine. The only good thing is that I briefly felt hope xD Aug 19, 2021 at 0:53
8
git clone --depth <Number> <repository> --branch <branch name> --single-branch

This command help me (Thanks to Nicola Paolucci)

for example

git clone --depth 1 https://github.com/gokhanmoral/siyahkernel3 --branch ics  --single-branch
1
  • I need only develop branch for now, so it helped!
    – Nike Kov
    Jul 29, 2020 at 8:30
5

If you have access to a 3rd-party server, you could clone there and then copy.

0
4

Use a git proxy, such as ngitcached or git-proxy.

2
  • 1
    Even better: github.com/git-cloner/gitcache Aug 7, 2020 at 11:03
  • this seems to still require we to have access to a server with an excellent connection to complete the inital cloning right? where such apps will be installed to do the initial download? I dont have it, and unreal engine is so huge... Aug 19, 2021 at 1:11
3

This problem bit me too. In my case there is a work-around. It may or may not apply in your case.

I'm using a mobile phone sometimes to initiate git operations on a remote system. If my wi-fi breaks of course the session ends and git drops the whole clone operation without recovering. But since the internet connection from my remote system to the git master is solid there's no need for the clone to stop. All I need is the commonsense to detach the clone from the terminal session. This can be done by using screen/tmux or nohup/daemon. So it's a liveware malfunction in my case.

3

Same problem here - I have a really flaky internet connection with often not more than 10-15 kb/sec :-P

For me the wget way worked very well.

Go to the repository site where the green button "clone or download" is, click it and copy the link of the ZIP download option.

Then insert the link to the wget command:

wget -c -m -np https://github.com/your/repository/archive/master.zip

Works like a charm...

1
  • Maybe this worked before, but right now when I try your solution and connection breaks (or I press Ctrl-C) then after rerun downloading is not continued but started from beginning, at least on LLVM repository.
    – Arty
    Aug 15, 2021 at 9:43
2

Use CNTRL Z to stop the cloning. Don't close the terminal put the system/laptop in hibernation and then continue later by fg command. I was facing this same problem today while trying to clone a repo frm github. This came as a time saver for me.

0
0

if we assume server's have good band-wide (and you have a server) another answer is to:

  1. create your own server using Server-Side Git Wrapper's
  2. clone it in your server
  3. Zip it using Server-Side Zip Archiver's
  4. download it from and with Server-Side Resume support

but this only works with very basic Web-development experience ;) and also you need git.exe in your sever

0

I've had a lot of trouble getting repositories over a high latency satellite connection. I had no success with the various config options that are commonly suggested which seem to be repeated without any real explanation of why they ought to work. My bandwidth is a couple of Mb/s and is sufficient to download large files (hundreds of MB) in a reasonable amount of time, but it seems some other instability in the connection causes fetch to fail.

One solution that I haven't seen mentioned here is to try with SSH instead of HTTPS (in combination with other suggestions like shallow cloning). This has been a lot more successful for me in cases where HTTPS would reliably fail. I imagine most people reading this are trying to clone from Github i.e. try setting up a key and using git clone --depth=1 --no-tags [email protected]:organisation/repo.git

A backup solution is to clone the repository somewhere else; ideally your own server, but since many folks don't have access to that I've found Google Colab is very serviceable. This also works if SSH is blocked on your network:

!git clone --depth=1 --no-tags https://github.com/some/repo.git
!tar -czf repo.tar.gz repo

and then download the tarball via the file explorer in the browser. You could also copy to Google Drive, scp/rsync or even cloud storage if you have the means. Running git fetch --unshallow on the extracted tarball also seems to generally work.

-1

You can try to use mercurial with the hg-git extension.

If that doesn't work you can can use git fetch <commit-id> to fetch only parts of a remote git repository (you can fetch into an empty git repository, there is no need to create it with clone). But you might to correct the branch configuration (=create local and remote tracking branches) when you use this approach.

-1

The best workaround that worked for me:

I faced the same issue with a bad internet connection. So I came up with the following solution:

I created a small php file on my server to download the package as a zip file:

<?php
$url = "https://codeload.github.com/CocoaPods/Specs/zip/master";
file_put_contents("coco.zip", fopen($url, 'r'));
?>  

<a href="coco.zip">coco.zip</a>

Then download the zip file using any download manager that supports resume.

1
  • 3
    You don't need a server or PHP for this. curl -ococo.zip https://codeload.github.com/CocoaPods/Specs/zip/master
    – tripleee
    Dec 9, 2020 at 6:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.