79

I'm trying to get the content of this url "https://itunes.apple.com/in/genre/ios-business/id6000?mt=8" and its showing this error

Traceback (most recent call last):
  File "/home/preetham/Desktop/eg.py", line 17, in <module>
    page1 = requests.get(ap)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max retries exceeded with url: /in/app/adobe-reader/id469337564?mt=8 (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)

the code is

url="https://itunes.apple.com/in/genre/ios-business/id6000?mt=8"
page = requests.get(url)
tree = html.fromstring(page.text)
flist=[]
plist=[]
for i in range(0,100):
    app = tree.xpath("//div[@class='column first']/ul/li/a/@href")
    ap=app[0]
    page1 = requests.get(ap)

when I try the range with (0,2) it works but when I put the range in 100's it shows this error.

  • Shouldn't you use ivariable somewhere in the for ? – Laurent S. Apr 11 '14 at 12:58
  • you are like requesting the same app a 100 times. what is that for ? – njzk2 Apr 11 '14 at 13:00
  • I am using i in the rest of the code. I have not posted the entire code – user3446000 Apr 11 '14 at 13:06
  • I am not requesting for the same app 100 times. I am requesting for 100 different apps under the same category. – user3446000 Apr 11 '14 at 13:07
  • Looks like your DNS resolver is unable to resolve itunes.apple.com. Can you run dig itunes.apple.com at your command line and post the results here? – Thomas Orozco Apr 11 '14 at 13:11
-2

I got the same problem and found the easiest solution for my case. Try to replace this:

url="https://itunes.apple.com/in/genre/ios-business/id6000?mt=8"

With this:

url="http://itunes.apple.com/in/genre/ios-business/id6000?mt=8"

The difference is "s" in protocol. This solved my trouble.

3

It is always good to implement exception handling. It does not only help to avoid unexpected exit of script but can also help to log errors and info notification. When using Python requests I prefer to catch exceptions like this:

    try:
        res = requests.get(adress,timeout=30)
    except requests.ConnectionError as e:
        print("OOPS!! Connection Error. Make sure you are connected to Internet. Technical Details given below.\n")
        print(str(e))            
        renewIPadress()
        continue
    except requests.Timeout as e:
        print("OOPS!! Timeout Error")
        print(str(e))
        renewIPadress()
        continue
    except requests.RequestException as e:
        print("OOPS!! General Error")
        print(str(e))
        renewIPadress()
        continue
    except KeyboardInterrupt:
        print("Someone closed the program")

Here renewIPadress() is a user define function which can change the IP address if it get blocked. You can go without this function.

21

Just do this,

Paste the following code in place of page = requests.get(url):

import time

page = ''
while page == '':
    try:
        page = requests.get(url)
        break
    except:
        print("Connection refused by the server..")
        print("Let me sleep for 5 seconds")
        print("ZZzzzz...")
        time.sleep(5)
        print("Was a nice sleep, now let me continue...")
        continue

You're welcome :)

  • 2
    remember to do import time – Yuan Tao Jul 21 '17 at 3:29
  • 3
    requests has his own code to handle its error and retry – Zulu Nov 24 '17 at 14:03
  • 3
    It never exits from the loop. @jatin – alper Jan 15 at 18:18
  • 3
    Also, not a good idea to just catch any type of exception (with except: ...) from requests and sleep() in response. Instead, they should catch requests.exceptions.ConnectionError and sleep() only if that exception occurs. (Or better yet, just use the builtin Retry() class that comes with requests, as suggested by @Zulu). – J. Taylor May 9 at 4:03
38

Just use requests' features:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry


session = requests.Session()
retry = Retry(connect=3, backoff_factor=0.5)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

session.get(url)

This will GET the URL and retry 3 times in case of requests.exceptions.ConnectionError. backoff_factor will help to apply delays between attempts to avoid to fail again in case of periodic request quota.

Take a look at requests.packages.urllib3.util.retry.Retry, it has many options to simplify retries.

  • 2
    Discovered so many new features here. – PascalVKooten Apr 2 at 21:18
13

pip install pyopenssl seemed to solve it for me.

https://github.com/requests/requests/issues/4246

  • 1
    awesome, it worked like a charm – user1742919 Dec 16 '17 at 8:56
91

What happened here is that itunes server refuses your connection (you're sending too many requests from same ip address in short period of time)

Max retries exceeded with url: /in/app/adobe-reader/id469337564?mt=8

error trace is misleading it should be something like "No connection could be made because the target machine actively refused it".

There is an issue at about python.requests lib at Github, check it out here

To overcome this issue (not so much an issue as it is misleading debug trace) you should catch connection related exceptions like so:

try:
    page1 = requests.get(ap)
except requests.exceptions.ConnectionError:
    r.status_code = "Connection refused"

Another way to overcome this problem is if you use enough time gap to send requests to server this can be achieved by sleep(timeinsec) function in python (don't forget to import sleep)

from time import sleep

All in all requests is awesome python lib, hope that solves your problem.

  • The sleep loop fixed my problem - a bit of a hack, but by looping a couple of times while handling the error response, I was able to brute force a solution. – elPastor Mar 29 '17 at 0:20
  • 5
    This answer is actually wrong. This is a resolver lookup issue, as indicated by the (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known) part. "gai" stands for getaddrinfo, and the probable related error is: EAI_NONAME The node or service is not known; or both node and service are NULL; or AI_NUMERICSERV was specified in hints.ai_flags and service was not a numeric port-number string. It probably looked like the sleep fixed it, but you probably just slept through a transient DNS resolver issue. – lingfish Jun 6 '17 at 23:13
  • This answer does not seem to make sense as in 'r' is the object that comes from requests.get() so with the exception this just leads to another error. – mikkokotila May 18 at 8:19

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.