13

On Ubuntu, I am trying to download a file (from a script) using wget. Buildling a program to download this file everyday and load to a hadoop cluster.

however, the wget fails, with the following message.

wget http://www.nseindia.com/content/historical/EQUITIES/2012/JUN/cm15JUN2012bhav.csv.zip
--2012-06-16 03:37:30--  http://www.nseindia.com/content/historical/EQUITIES/2012/JUN/cm15JUN2012bhav.csv.zip
Resolving www.nseindia.com... 122.178.225.48, 122.178.225.18
Connecting to www.nseindia.com|122.178.225.48|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2012-06-16 03:37:30 ERROR 403: Forbidden.

when I try the same url in firefox or equivalent, it works just fine. And yes, there is no license agreement kind of thing involved...

Am I missing something basic regarding wget ??

2
  • how far back in time can you fetch that data with wget? I assume you are constructing the URLs for each trading day by concatenating the url strings? Curious to know. Aug 22, 2012 at 21:17
  • Well, I believe, NSEIndia has data going back till 2000 or so... bSEIndia has similar service, and they go back in time even further...
    – Raghav
    Mar 27, 2013 at 16:40

5 Answers 5

15

The site blocks wget because wget uses an uncommon user-agent by default. To use a different user-agent in wget, try:

wget -U Mozilla/5.0 http://www.nseindia.com/content/historical/EQUITIES/2012/JUN/cm15JUN2012bhav.csv.zip
5
  • That is not completely true. It has an user-agent: Wget/VERSION according to wget --help.
    – Zagorax
    Jun 15, 2012 at 22:19
  • wow, worked like a charm.. thanks a ton. Probably being ambitious now, isnt there a bulk get kind of version, because with this wget now, i dont get through trying to do a *.zip or similar... any tips there...
    – Raghav
    Jun 15, 2012 at 22:20
  • 1
    Try wget --help, there's an option for recursive download and another one to list extension you want to download.
    – Zagorax
    Jun 15, 2012 at 22:25
  • 2
    @learn4living, To download multiple URLs in a single wget command, I usually do wget -U Mozilla/5.0 "http://www.example.com/file1.zip" "http://www.example.com/file2.zip"
    – enderskill
    Jun 15, 2012 at 22:28
  • That did not work for me. On the other hand, using curl instead of wget as suggested by @anas worked... still don't understand exactly why, though...
    – dokaspar
    Jun 23, 2016 at 6:25
6

Use:

wget -U mozilla http://www.nseindia.com/content/historical/EQUITIES/2012/JUN/cm15JUN2012bhav.csv.zip

Some sites simply prevent wget user-agent to download files. I just downloaded that file with this command. It works.

1

Another technique webapps or webservers may use is to check the 'Referrer' content header value. In addition to specifying the user agent, it may be necessary to supply the referrer url.

e.g.,

wget --referer http://freestockphotos.com/Scenery1.html http://freestockphotos.com/SKY/TreeSunset.jpg

This host appears to reject requests for the target file if they were not made while navigating from the 'Scenery1.html' page.

1

I use curl -O <URL> because wget don't support HTTPS and some other protocols.

1
  • wget supports https! From the manpage: GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies. Apr 28, 2019 at 21:26
0

Some sites simply prevent wget user-agent to download files wget -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' http://yourURL.com

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.