A page contains links to a set of .zip files, all of which I want to download. I know this can be done by wget and curl. How is it done?
3 Answers
The command is:
wget -r -np -l 1 -A zip http://example.com/download/
Options meaning:
-r, --recursive specify recursive download.
-np, --no-parent don't ascend to the parent directory.
-l, --level=NUMBER maximum recursion depth (inf or 0 for infinite).
-A, --accept=LIST comma-separated list of accepted extensions.
-
16The
-nd
(no directories) flag is handy if you don't want any extra directories created (i.e., all files will be in the root folder). Nov 6, 2013 at 23:19 -
1How do I tweak this solution for it to go deeper from the given page? I tried -l 20, but wget stops immediatly.– WrenchNov 27, 2015 at 14:46
-
2If the files aren't in the same directory as the starting URL, you might need to get rid of
-np
. If they're on a different host, you'll need--span-host
.– DanSep 26, 2018 at 15:28 -
Is there a way to keep the directory structure of the website, but exclude the root folder only, such that the current directly is the root folder of the website instead of a folder with the name of the website's URL? Jun 2, 2021 at 13:39
Above solution does not work for me. For me only this one works:
wget -r -l1 -H -t1 -nd -N -np -A.mp3 -erobots=off [url of website]
Options meaning:
-r recursive
-l1 maximum recursion depth (1=use only this directory)
-H span hosts (visit other hosts in the recursion)
-t1 Number of retries
-nd Don't make new directories, put downloaded files in this one
-N turn on timestamping
-A.mp3 download only mp3s
-erobots=off execute "robots.off" as if it were a part of .wgetrc
-
2
-
yes, thanks! I didn't remember where it came from, have it just lying in my scripts. Sep 18, 2014 at 19:33
-
-
1+1 for the
-H
switch. This is what was preventing the first answer (which is what I tried before looking on SO) from working.– AlexJun 26, 2017 at 16:10 -
1
For other scenarios with some parallel magic I use:
curl [url] | grep -i [filending] | sed -n 's/.*href="\([^"]*\).*/\1/p' | parallel -N5 wget -