How can I resume pull
when disconnected? The pull
process always start from the beginning every time I run docker pull some-image
again after disconnected. My connection is so unstable that even downloading just a 100MB image take so long and almost fails every time. So, it is almost impossible for me to pull a bigger image. So, how can I resume the pull process?
4 Answers
Update:
The pull
process will now automatically resume based on which layers have already been downloaded. This was implemented with https://github.com/moby/moby/pull/18353.
Old:
There is no resume
feature yet. However there are discussions around this feature being implemented with docker's download manager.
-
2It would be real nice to have a resume partial download like wget. I assume sha hashes are maintained to verify integrity. I am setting up a virtual "develoipment workstation" as a docker image and some of the team members are on cellular modems out in the boonies and are having a hard time with 800meg layers. Though originally intended for server clouds connected by fiber "The street finds it's own uses for things" and it could end up evolving to a form of OS kernel for running virtualized apps.on workstations and other devices.– peterkOct 26, 2018 at 3:03
-
Your comment might bring more benefit at the github discussion linked my NReilingh. Oct 26, 2018 at 11:54
-
Feb 2020, docker pull downloads the layers again when interrupted. Feb 11, 2022 at 16:45
-
2023 and still downloading the layers again. I have one layer with 2.7GB to download via unstable VPN and with docker this seems unfeasible. I have to download manually with a download manager that can resume and then somehow feed this manually into docker...– JörgJun 19 at 15:34
-
The feature is still experimental and needs to be enabled manually: user-images.githubusercontent.com/799078/…– JörgJun 19 at 16:01
Docker's code isn't as updated as the moby in development repository on github. People have been having issues for several years relating to this. I had tried to manually use several patches which aren't in the upstream yet, and none worked decent.
The github repository for moby (docker's development repo) has a script called download-frozen-image-v2.sh. This script uses bash, curl, and other things like JSON interpreters via command line. It will retrieve a docker token, and then download all of the layers to a local directory. You can then use 'docker load' to insert into your local docker installation.
It does not do well with resume though. It had some comment in the script relating to 'curl -C' isn't working. I had tracked down, and fixed this problem. I made a modification which uses a ".headers" file to retrieve initially, which has always returned a 302 while I've been monitoring, and then retrieves the final using curl (+ resume support) to the layer tar file. It also has to loop on the calling function which retrieves a valid token which unfortunately only lasts about 30 minutes.
It will loop this process until it receives a 416 stating that there is no resume possible since it's ranges have been fulfilled. It also verifies the size against a curl header retrieval. I have been able to retrieve all images necessary using this modified script. Docker has many more layers relating to retrieval, and has remote control processes (Docker client) which make it more difficult to control, and they viewed this issue as only affecting some people on bad connections.
I hope this script can help you as much as it has helped me:
Changes: fetch_blob function uses a temporary file for its first connection. It then retrieves 30x HTTP redirect from this. It attempts a header retrieval on the final url and checks whether the local copy has the full file. Otherwise, it will begin a resume curl operation. The calling function which passes it a valid token has a loop surrounding retrieving a token, and fetch_blob which ensures the full file is obtained.
The only other variation is a bandwidth limit variable which can be set at the top, or via "BW:10" command line parameter. I needed this to allow my connection to be viable for other operations.
Click here for the modified script.
In the future it would be nice if docker's internal client performed resuming properly. Increasing the amount of time for the token's validation would help tremendously..
Brief views of change code:
#loop until FULL_FILE is set in fetch_blob.. this is for bad/slow connections
while [ "$FULL_FILE" != "1" ];do
local token="$(curl -fsSL "$authBase/token?service=$authService&scope=repository:$image:pull" | jq --raw-output '.token')"
fetch_blob "$token" "$image" "$layerDigest" "$dir/$layerTar" --progress
sleep 1
done
Another section from fetch_blob:
while :; do
#if the file already exists.. we will be resuming..
if [ -f "$targetFile" ];then
#getting current size of file we are resuming
CUR=`stat --printf="%s" $targetFile`
#use curl to get headers to find content-length of the full file
LEN=`curl -I -fL "${curlArgs[@]}" "$blobRedirect"|grep content-length|cut -d" " -f2`
#if we already have the entire file... lets stop curl from erroring with 416
if [ "$CUR" == "${LEN//[!0-9]/}" ]; then
FULL_FILE=1
break
fi
fi
HTTP_CODE=`curl -w %{http_code} -C - --tr-encoding --compressed --progress-bar -fL "${curlArgs[@]}" "$blobRedirect" -o "$targetFile"`
if [ "$HTTP_CODE" == "403" ]; then
#token expired so the server stopped allowing us to resume, lets return without setting FULL_FILE and itll restart this func w new token
FULL_FILE=0
break
fi
if [ "$HTTP_CODE" == "416" ]; then
FULL_FILE=1
break
fi
sleep 1
done
I am on Arch Linux with docker 24.0.2 (not docker desktop, just whatever you get with pacman -S docker
).
You can enable docker pull
to resume interrupted downloads by adding "features": {"containerd-snapshotter": true}
to /etc/docker/daemon.json
.
sudo mkdir /etc/docker
sudo gedit /etc/docker/daemon.json
Then make it look like:
{
"features": {"containerd-snapshotter": true}
}
and restart dockerd:
systemctl restart docker
Please note that this feature is still kind of experimental and may change in the future. Also, it seems like it enables some other stuff as well, like downloading a lot of files at once. That works fine on my machine, but if your internet connection is slow, you may have problems.
Try this
ps -ef | grep docker
Get PID of all the docker pull
command and do a kill -9
on them. Once killed, re-issue the docker pull <image>:<tag>
command.
This worked for me!
-
Is this any different from just
Ctrl-C
on the current docker stable version? This may allow you to pull an image without re-downloading already completed layers, but this Q is about resuming downloads on layers that were only partially downloaded before stalling. Feb 8, 2018 at 0:40 -
Ctrl-C
killed the docker session on my terminal but I could still see some docker related zombie processes. My answer is more focused on a cleaner approach to re-pull the image. I'm not sure why this was down voted. Mar 29, 2018 at 22:27 -
Did not work for me. Also did not work: keeping the hung terminal, opening another terminal and issuing the command again. This resulted in an identical status, all completed except the two downloads that were hanging from the first one, still hung on the second one. Aug 19, 2019 at 17:44