44

I want to use wget to download the following 18 HTML files:

http://www.ted.com/talks/quick-list?sort=date&order=desc&page=18
http://www.ted.com/talks/quick-list?sort=date&order=desc&page=17
...
http://www.ted.com/talks/quick-list?sort=date&order=desc&page=1

No matter what comes after page=, it always downloads the first page of the listing. Do I have to escape some characters in the URLs? How?

6
  • Have you tried this ? ted.com/talks/quick-list?sort=date&desc&page=17
    – saruftw
    Oct 20, 2014 at 21:13
  • 2
    Are you doing wget "http://www.ted.com/talks/quick-list?sort=date&order=desc&page=18" or not using the double-quotes?
    – hrbrmstr
    Oct 20, 2014 at 21:18
  • @rgbimbochamp I'd give it a try. Is there an explanation why should that fix the problem though? Oct 20, 2014 at 21:23
  • @hrbrmstr ahh! so I should wrap the whole url in double quotations? I'll try that. Oct 20, 2014 at 21:24
  • & is a special character in most shell environments, so it never makes it to wget as it executes the call in the background. If this works, I'll post it as an answer.
    – hrbrmstr
    Oct 20, 2014 at 21:26

3 Answers 3

81

& is a special character in most shell environments. You can use double quotes to quote the URL to pass the whole thing in as the parameter to wget:

wget "http://www.ted.com/talks/quick-list?sort=date&order=desc&page=18"
1
  • 1
    How would you use the {1..20} to download a range of files using a string?
    – Neil
    Mar 7, 2016 at 16:41
5
  1. Store your list of URLs in a file (each URL in a separate line!!):

    echo "http://www.ted.com/talks/quick-list?sort=date&order=desc&page=18 http://www.ted.com/talks/quick-list?sort=date&order=desc&page=17 ... " > wget_filelist.txt

  2. Call wget to retrieve the stuff:

    wget -i wget_filelist.txt

2

Special case: There is still a problem with wget "URL" format, even though it solved the problem of & it can't pass ! symbol.

Solution: Single quote instead of double quote for the URL will fix this, example:

wget 'https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-G.798-201712-I!!PDF-E&type=items'

As shown in above example, it works for me which contains both & and ! symbols in it. I am not sure if it's an excepted solution for all platform (ie., official POSIX shell).

Bonus: Further we can use wget -c 'URL', so that, in case there is a failure in one go and we don't need to start from beginning.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.