26

How can I resume pull when disconnected? The pull process always start from the beginning every time I run docker pull some-image again after disconnected. My connection is so unstable that even downloading just a 100MB image take so long and almost fails every time. So, it is almost impossible for me to pull a bigger image. So, how can I resume the pull process?

4 Answers 4

20

Update:

The pull process will now automatically resume based on which layers have already been downloaded. This was implemented with https://github.com/moby/moby/pull/18353.

Old:

There is no resume feature yet. However there are discussions around this feature being implemented with docker's download manager.

5
  • 2
    It would be real nice to have a resume partial download like wget. I assume sha hashes are maintained to verify integrity. I am setting up a virtual "develoipment workstation" as a docker image and some of the team members are on cellular modems out in the boonies and are having a hard time with 800meg layers. Though originally intended for server clouds connected by fiber "The street finds it's own uses for things" and it could end up evolving to a form of OS kernel for running virtualized apps.on workstations and other devices.
    – peterk
    Oct 26, 2018 at 3:03
  • Your comment might bring more benefit at the github discussion linked my NReilingh. Oct 26, 2018 at 11:54
  • Feb 2020, docker pull downloads the layers again when interrupted. Feb 11, 2022 at 16:45
  • 2023 and still downloading the layers again. I have one layer with 2.7GB to download via unstable VPN and with docker this seems unfeasible. I have to download manually with a download manager that can resume and then somehow feed this manually into docker...
    – Jörg
    Jun 19 at 15:34
  • The feature is still experimental and needs to be enabled manually: user-images.githubusercontent.com/799078/…
    – Jörg
    Jun 19 at 16:01
4

Docker's code isn't as updated as the moby in development repository on github. People have been having issues for several years relating to this. I had tried to manually use several patches which aren't in the upstream yet, and none worked decent.

The github repository for moby (docker's development repo) has a script called download-frozen-image-v2.sh. This script uses bash, curl, and other things like JSON interpreters via command line. It will retrieve a docker token, and then download all of the layers to a local directory. You can then use 'docker load' to insert into your local docker installation.

It does not do well with resume though. It had some comment in the script relating to 'curl -C' isn't working. I had tracked down, and fixed this problem. I made a modification which uses a ".headers" file to retrieve initially, which has always returned a 302 while I've been monitoring, and then retrieves the final using curl (+ resume support) to the layer tar file. It also has to loop on the calling function which retrieves a valid token which unfortunately only lasts about 30 minutes.

It will loop this process until it receives a 416 stating that there is no resume possible since it's ranges have been fulfilled. It also verifies the size against a curl header retrieval. I have been able to retrieve all images necessary using this modified script. Docker has many more layers relating to retrieval, and has remote control processes (Docker client) which make it more difficult to control, and they viewed this issue as only affecting some people on bad connections.

I hope this script can help you as much as it has helped me:

Changes: fetch_blob function uses a temporary file for its first connection. It then retrieves 30x HTTP redirect from this. It attempts a header retrieval on the final url and checks whether the local copy has the full file. Otherwise, it will begin a resume curl operation. The calling function which passes it a valid token has a loop surrounding retrieving a token, and fetch_blob which ensures the full file is obtained.

The only other variation is a bandwidth limit variable which can be set at the top, or via "BW:10" command line parameter. I needed this to allow my connection to be viable for other operations.

Click here for the modified script.

In the future it would be nice if docker's internal client performed resuming properly. Increasing the amount of time for the token's validation would help tremendously..

Brief views of change code:

#loop until FULL_FILE is set in fetch_blob.. this is for bad/slow connections
            while [ "$FULL_FILE" != "1" ];do
                local token="$(curl -fsSL "$authBase/token?service=$authService&scope=repository:$image:pull" | jq --raw-output '.token')"
                fetch_blob "$token" "$image" "$layerDigest" "$dir/$layerTar" --progress
                sleep 1
            done

Another section from fetch_blob:

while :; do
            #if the file already exists.. we will be resuming..
            if [ -f "$targetFile" ];then
                #getting current size of file we are resuming
                CUR=`stat --printf="%s" $targetFile`
                #use curl to get headers to find content-length of the full file
                LEN=`curl -I -fL "${curlArgs[@]}" "$blobRedirect"|grep content-length|cut -d" " -f2`

                #if we already have the entire file... lets stop curl from erroring with 416
                if [ "$CUR" == "${LEN//[!0-9]/}" ]; then
                    FULL_FILE=1
                    break
                fi
            fi

            HTTP_CODE=`curl -w %{http_code} -C - --tr-encoding --compressed --progress-bar -fL "${curlArgs[@]}" "$blobRedirect" -o "$targetFile"`
            if [ "$HTTP_CODE" == "403" ]; then
                #token expired so the server stopped allowing us to resume, lets return without setting FULL_FILE and itll restart this func w new token
                FULL_FILE=0
                break
            fi

            if [ "$HTTP_CODE" == "416" ]; then
                FULL_FILE=1
                break
            fi

            sleep 1
        done
2

I am on Arch Linux with docker 24.0.2 (not docker desktop, just whatever you get with pacman -S docker).

You can enable docker pull to resume interrupted downloads by adding "features": {"containerd-snapshotter": true} to /etc/docker/daemon.json.

sudo mkdir /etc/docker
sudo gedit /etc/docker/daemon.json

Then make it look like:

{
    "features": {"containerd-snapshotter": true}
}

and restart dockerd:

systemctl restart docker

Please note that this feature is still kind of experimental and may change in the future. Also, it seems like it enables some other stuff as well, like downloading a lot of files at once. That works fine on my machine, but if your internet connection is slow, you may have problems.

Link to config file documentation.

-4

Try this

ps -ef | grep docker

Get PID of all the docker pull command and do a kill -9 on them. Once killed, re-issue the docker pull <image>:<tag> command.

This worked for me!

3
  • Is this any different from just Ctrl-C on the current docker stable version? This may allow you to pull an image without re-downloading already completed layers, but this Q is about resuming downloads on layers that were only partially downloaded before stalling.
    – NReilingh
    Feb 8, 2018 at 0:40
  • Ctrl-C killed the docker session on my terminal but I could still see some docker related zombie processes. My answer is more focused on a cleaner approach to re-pull the image. I'm not sure why this was down voted.
    – underwood
    Mar 29, 2018 at 22:27
  • Did not work for me. Also did not work: keeping the hung terminal, opening another terminal and issuing the command again. This resulted in an identical status, all completed except the two downloads that were hanging from the first one, still hung on the second one. Aug 19, 2019 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.