I want to crawl a site, however cloudflare was getting in the way. I was able to get the servers IP, so cloudflare won't bother me.

How can I utilize this in the requests library?

For example, I want to go directly to www.example.com/foo.php, but in requests it will resolve the IP on the cloudflare network instead of the one I want it to use. How can I make it use the one I want it to use?

I would of sent in a request so the real IP with the host set as the www.example.com, but that will just give me the home page. How can I visit other links on the site?

up vote 10 down vote accepted

You will have to set a custom header host with value of example.com, something like:

requests.get('http://127.0.0.1/foo.php', headers={'host': 'example.com'})

should do the trick. If you want to verify that then type in the following command (requires netcat): nc -l -p 80 and then run the above command. It will produce output in the netcat window:

GET /foo.php HTTP/1.1
Host: example.com
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.2 CPython/3.4.3 Windows/8
  • Works, thank you :) – user3893623 May 1 '15 at 21:45
  • 1
    Works only for http. If you do that with https, you'll get an error that the hostname doesn't match the certificate. – Patrick Valsecchi Feb 25 '17 at 9:23
  • @tymoteusz-paul You can disable SSL certficate verification in Requests, that should allow you to have access to the server but open you up to man in the middle attacks: stackoverflow.com/questions/15445981/… – AceLewis Oct 11 '17 at 10:10

You'd have to tell requests to fake the Host header, and replace the hostname in the URL with the IP address:

requests.get('http://123.45.67.89/foo.php', headers={'Host': 'www.example.com'})

The URL 'patching' can be done with the urlparse library:

parsed = urlparse.urlparse(url)
hostname = parsed.hostname
parsed = parsed._replace(netloc=ipaddress)
ip_url = parsed.geturl()

response = requests.get(ip_url, headers={'Host': hostname})

Demo against Stack Overflow:

>>> import urlparse
>>> import socket
>>> url = 'http://stackoverflow.com/help/privileges'
>>> parsed = urlparse.urlparse(url)
>>> hostname = parsed.hostname
>>> hostname
'stackoverflow.com'
>>> ipaddress = socket.gethostbyname(hostname)
>>> ipaddress
'198.252.206.16'
>>> parsed = parsed._replace(netloc=ipaddress)
>>> ip_url = parsed.geturl()
>>> ip_url
'http://198.252.206.16/help/privileges'
>>> response = requests.get(ip_url, headers={'Host': hostname})
>>> response
<Response [200]>

In this case I looked up the ip address dynamically.

  • Works great. Unfortunately there seems to be a bug that when you use POST and a data generator (for chunked encoding), you end up having two 'Host' headers, the original AND the new one. :( – jlh Nov 20 '17 at 14:13
  • @jlh: if you have a simple reproducible case I can take a look if that can be fixed. – Martijn Pieters Nov 20 '17 at 14:16
  • Sure, here's an example: pastebin.com/2WQFWQit – jlh Nov 20 '17 at 14:28
  • @jlh: looks like a bug; a different low-level path through the HTTP library is taken and it is not told to skip the host header. – Martijn Pieters Nov 20 '17 at 18:39
  • @jlh: filed issue #4392. – Martijn Pieters Nov 20 '17 at 19:04

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.