0

I would like to download all pdf files linked on a website using wget on Mac OS (zsh).

I have tried:

wget -r -p -k --random-wait --limit-rate=50k -A .pdf -erobots=off https://unfccc.int/process-and-meetings/bodies/constituted-bodies/executive-committee-of-the-warsaw-international-mechanism-for-loss-and-damage-wim-excom/task-force-on-displacement/implementation-updates-task-force-on-displacement\#eq-1

and I have also added the following options to no avail:

  • --span-hosts
  • --no-check-certificate
  • --no-cookies
  • -H

The error is always the same:

no-follow attribute found in unfccc.int/process-and-meetings/bodies/constituted-bodies/executive-committee-of-the-warsaw-international-mechanism-for-loss-and-damage-wim-excom/task-force-on-displacement/implementation-updates-task-force-on-displacement. Will not follow any links on this page

1
  • Shouldn't it be -e robots=off not -erobots=off? Although it doesn't work either way for me. This may help bitbucket.org/intgr/pub/wiki/… But I myself stumbled on downloading the correct html.
    – x00
    Jul 11, 2020 at 15:34

1 Answer 1

0

First make sure you has permission to crawl the pages, I'm not gonna be responsible for bad thing happend to you or anyone after bypassing the robot no-follow attribute !

The tricks to ignore all restrictions and be a bad crawler bot you just need to include :
-e robots=off on your command.

Yes, you've included it on your commands but you got some typos ! Just like @x00 say it need to be -e robots=off instead of -erobots=off.

I Don't know why @x00 doesn't answer the question with his answer :/

Note : For x00, if you want to me to delete my answer because its nearly similar with yours comment , just command under my answer and i will delete it for you anytime !

1
  • From my comment: "Although it doesn't work either way for me". That's one but not the only reason why I didn't write an answer.
    – x00
    Jul 21, 2020 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.