86

A page contains links to a set of .zip files, all of which I want to download. I know this can be done by wget and curl. How is it done?

0

3 Answers 3

136

The command is:

wget -r -np -l 1 -A zip http://example.com/download/

Options meaning:

-r,  --recursive          specify recursive download.
-np, --no-parent          don't ascend to the parent directory.
-l,  --level=NUMBER       maximum recursion depth (inf or 0 for infinite).
-A,  --accept=LIST        comma-separated list of accepted extensions.
4
  • 16
    The -nd (no directories) flag is handy if you don't want any extra directories created (i.e., all files will be in the root folder). Nov 6, 2013 at 23:19
  • 1
    How do I tweak this solution for it to go deeper from the given page? I tried -l 20, but wget stops immediatly.
    – Wrench
    Nov 27, 2015 at 14:46
  • 2
    If the files aren't in the same directory as the starting URL, you might need to get rid of -np. If they're on a different host, you'll need --span-host.
    – Dan
    Sep 26, 2018 at 15:28
  • Is there a way to keep the directory structure of the website, but exclude the root folder only, such that the current directly is the root folder of the website instead of a folder with the name of the website's URL? Jun 2, 2021 at 13:39
91

Above solution does not work for me. For me only this one works:

wget -r -l1 -H -t1 -nd -N -np -A.mp3 -erobots=off [url of website]

Options meaning:

-r            recursive
-l1           maximum recursion depth (1=use only this directory)
-H            span hosts (visit other hosts in the recursion)
-t1           Number of retries
-nd           Don't make new directories, put downloaded files in this one
-N            turn on timestamping
-A.mp3        download only mp3s
-erobots=off  execute "robots.off" as if it were a part of .wgetrc
9
  • 2
    Source: commandlinefu.com/commands/view/12498/… Sep 18, 2014 at 16:10
  • yes, thanks! I didn't remember where it came from, have it just lying in my scripts. Sep 18, 2014 at 19:33
  • don't know sorry. make a new question! ;) Feb 12, 2015 at 21:36
  • 1
    +1 for the -H switch. This is what was preventing the first answer (which is what I tried before looking on SO) from working.
    – Alex
    Jun 26, 2017 at 16:10
  • 1
    Nope, you answered this on 2013-09-10.
    – Quasímodo
    Mar 15, 2021 at 20:11
6

For other scenarios with some parallel magic I use:

curl [url] | grep -i [filending] | sed -n 's/.*href="\([^"]*\).*/\1/p' |  parallel -N5 wget -

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.