1

I need to scan a given set of URLs and get the HTTP status codes like "200 OK" in Python. I'm currently using urllib to do this. Is there a faster way to do this?

Python code

def get_status(url):
try:
    return urllib.urlopen(url).getcode()

except StandardError :
    return None

5 Answers 5

4

Couple remarks I would make for faster happier status checking. The first tip would be to use the http HEAD method. This asks the server for just the http headers (including the status code) without having it also serve the body of the page.

Second urllib works but I would recommend using the wonderful Requests library which provides a much nicer api for pretty much everything you would want to do with http.

Last I would use the gevents library to enable you to download each header asynchronously, vastly speeding up the whole process.

2
  • I think the http HEAD method will do the job.And thanks for the info about other libs..I probably should have a look about gevents.
    – Madushan
    Jul 19, 2012 at 17:35
  • Sharp recommendation on using HEAD over GET. Smart.
    – Jeff
    Jul 19, 2012 at 17:53
2

You probably want to do it in parallel in a non-blocking way. Check out eventlet library here: http://eventlet.net/. You can just grab an example from the front page http://eventlet.net/#web-crawler-example.

4
  • As i can understand,the lib will lode the whole page before it gives the status code.is that correct?
    – Madushan
    Jul 19, 2012 at 17:40
  • The example, yes; but you can easily see what kind of change you have to make.
    – cababunga
    Jul 19, 2012 at 17:47
  • I'v look in to the documentation.think that may help.do you think the libb is available via easy_install?
    – Madushan
    Jul 19, 2012 at 17:54
  • Same front page says: easy_install eventlet.
    – cababunga
    Jul 19, 2012 at 18:00
2

For speed, try using GRequests to check urls asynchronously (not one at a time).

Code

import grequests

urls = [
    'http://www.heroku.com',
    'http://tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

rs = (grequests.get(u) for u in urls)
# For even faster status code checks, use the HEAD method instead of GET
# rs = (grequests.head(u) for u in urls)

for r in grequests.map(rs):
    print r.status_code, r.url

Output

200 http://www.heroku.com/
200 http://tablib.org/
200 http://httpbin.org/
200 http://docs.python-requests.org/en/latest/index.html
200 http://kennethreitz.com/
4
  • Is this lib available via easy_install?..i'm using openshift as the pass and i can't directly install any lib.
    – Madushan
    Jul 19, 2012 at 17:28
  • I would recommend using pip over easy_install locally. You can easily add dependencies to an openshift project: openshift.redhat.com/community/kb/…
    – Jeff
    Jul 19, 2012 at 17:49
  • thanx for the info.don't you know about any lib that would scan a given array of urls asynchronously? with out me tell to do it?
    – Madushan
    Jul 19, 2012 at 18:02
  • Sorry, no. Not that I know of.
    – Jeff
    Jul 19, 2012 at 18:06
1

Yes there is.

  1. Use multiple threads all checking different URLs at the same time.
  2. Use raw sockets implementing a simple HTTP request. As soon as you get the 200 response (or any other code) you close the connection, avoiding unnecessary transfer of data.
1
  • I can't practically use threads since i'm writing wsgi app.please let me know if i can use them without constructing a global object,to directly out put the result of the scan to a webpage. I'm currently working on the 2nd method.
    – Madushan
    Jul 19, 2012 at 17:50
0

Use threading. Put your code in a Thread class, and store the results in a global object. Call a bunch of threads.

1
  • Yes i could use threading.but the problem is i'm using this function in a wsgi application to scan like 1000++ urls.if i use a global object to store the results.i have to wait untill the whole scan finished.and the web site i'm willing to write will give me a gateway time out.
    – Madushan
    Jul 19, 2012 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.