0

I visited this site: http://www.avcodes.co.uk/airlcodesearch.asp

And, selected the last option:

Select a letter for ICAO Codes: and chose "B"

Then click Submit.

I monitored the progress using using Tamper Data and Live HTTP Headers from Firefox.

And, all is well.. and the required direct URL to achieve the same effect is this:

http://www.avcodes.co.uk/airllistres.asp?statuslst=Y&iataairllst=&icaoairllst=B&B1=Submit

However, when using the above URL.. the data is NOT returned.

What am I missing and how can I find the correct URL.

The objective of this exercise is once I know the URL.. I will use a python script to loop through A to Z and get at the content from all the pages.

Please help.

3 Answers 3

6

I assume that you have permission from the website to use their database and that you are allowed to scrape their website. In any other case, it may be illegal to do this, depending on jurisdiction.

The problem here is that you are using GET to retrieve the contents, but the website is expecting a POST. Get and POST are not equivalent, although some programmers consider them the same (in PHP, for example, you can use $_REQUEST instead of $_GET and $_POST). This website is not like them, so you have to POST to this website.

In Python, you can post data to an url like this:

import urllib2
u = urllib2.urlopen("http://www.avcodes.co.uk/airllistres.asp", "statuslst=Y&iataairllst=&icaoairllst=B&B1=Submit")
print u.read()
2
  • Thanks.. that works!! Why would the get work on some webisite which expects a post by doing the method i highlighted in the question. So, you are saying that I can't rely on this... right?
    – ihightower
    Jun 6, 2012 at 15:59
  • Some Web apps are written to accept values in the query string or in a POST rquest body interchangeably; some are not. If an app only looks for vaules in one place or the other, you have to pass them where it expects.
    – kindall
    Jun 6, 2012 at 16:13
2

POST and GET are not equivalent. If the site is written to only accept GET, that's all that will work.

However you can just as easily write a script that uses POST.

1

That's because you need to use the POST method on that page.

Try this:

pip install requests

Then:

>>> import requests
>>> r = requests.post('http://www.avcodes.co.uk/airllistres.asp?statuslst=Y&iataairllst=&icaoairllst=B&B1=Submit')
>>> r.text

However it looks like there's post data you'll need to figure out in addition to the above query string arguments to get this to really return data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.