37

I want to curl to download a link, but I want it to skip files that already exist. Right now, the line of code I have will continue to overwrite it no mater what:

curl '$url' -o /home/$outputfile &>/dev/null &

How this can be achieved?

5 Answers 5

37

You can use curl option -C -. This option is used to resume a broken download, but will skip the download if the file is already complete. Note that the argument to -C is a single dash. A disadvantage might be that curl still briefly contacts the remote server to ask for the file size.

3
  • 7
    Can't download file (file does not exist) if the server doesn't support to resume. Error: curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume.
    – Dylan B
    Apr 10, 2018 at 13:43
  • @AbdennourTOUMI Try github.com/curl/curl
    – Wolfgang
    Jan 21, 2022 at 15:26
  • Doesn't work with -J: curl: --continue-at and --remote-header-name cannot be combined
    – Navin
    Feb 20 at 18:52
31

You could just put your call to curl inside an if block:

if ! [ -f /home/$outputfile ]; then
  curl -o /home/$outputfile "url"
fi

Also note that in your example, you've got $url inside single quotes, which won't do what you want. Compare:

echo '$HOME'

To:

echo "$HOME"

Also, curl has a --silent option that can be useful in scripts.

2
  • very helpful, thank you for pointing that out. but I am doing this via php, how can i run that if statement via php?
    – thevoipman
    Aug 8, 2012 at 6:14
  • 1
    I don't know PHP off the top of my head, but I'm sure it has a relatively simple mechanism for seeing if a file exists. Aug 9, 2012 at 14:07
19

Use wget with --no-clobber instead:

-nc, --no-clobber: skip downloads that would download to existing files.

Example:

wget -nc -q -O "/home/$outputfile" "$url" 
5
  • 1
    can you please guide me on how to use with wget?
    – thevoipman
    Aug 8, 2012 at 1:23
  • 4
    Initially I was using wget, but then I changed to curl because I wanted the newly downloaded file to have the timestamp of when it was downloaded. using wget, it does not generate new timestamp but instead whatever timestamp is from the source it copied. how do i have wget generating new timestamps?
    – thevoipman
    Aug 10, 2012 at 13:58
  • 1
    -N turns-off timestamping in wget. Feb 12, 2017 at 22:08
  • 1
    The -nc option will not download newer files. Use -N instead, it will download and overwrite the file if there's a newer version.
    – Noam Manos
    Jul 24, 2018 at 9:41
  • "use wget" doesn't seem to be the answer to "how do I do this with curl"... Feb 5 at 21:53
2

The curl may support skipping the files when you use it with -O and -J, but its behaviour is inconsistent.

The -J (--remote-header-name) basically tells the -O (--remote-name) option to use the server-specified Content-Disposition filename instead of extracting a filename from the URL. In that way the curl doesn't really know what file name the server will return, so it may ignore the existing file for a safety precaution.

Source: Re: -J "Refusing to overwrite..."

For example:

$ curl -LJO -H 'Accept: application/octet-stream' 'https://api.github.com/repos/x/y/releases/assets/12345
Warning: Refusing to overwrite my_file.bin: File 
Warning: exists
curl: (23) Failed writing body (0 != 16384)

However as mentioned already, its behaviour is unpredictable and it doesn't work for all the files.

2

As of version 7.83, released on 27th April 2022, curl now has a --no-clobber option, just like wget, which will do almost what you want.

2
  • 1
    Your post is here forever, so what is "recent"? Please update this to say as of which version that option exists. Feb 5 at 21:54
  • Nope, this just writes to filename.1 when filename exists. You'd think skipping existing files would be a really basic feature, but it's unsupported.
    – Navin
    Feb 20 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.