-1

I am trying to get all the 'a' tags which are used for links and also the the 'form' tag. The code that I have written fetches the whole page.

import requests
from requests.exceptions import HTTPError

for url in ['http://www.example.com', 'http://mail.example.com']:
    try:
        response = requests.get(url)

        # If the response was successful, no Exception will be raised
        response.raise_for_status()
    except HTTPError as http_err:
        print(f'HTTP error occurred: {http_err}')  # Python 3.6
    except Exception as err:
        print(f'Other error occurred: {err}')  # Python 3.6
    else:
        response.encoding = 'utf-8' # Optional: requests infers this internally
        print(response.text)

I can use regular expressions to get a specific thing from the page, but I don't know how to get the entire contents of a particular tag.

Thanks

2

1 Answer 1

3

You can use BeautifulSoup to parse html page:

import requests
from requests.exceptions import HTTPError
from bs4 import BeautifulSoup

for url in ['http://www.example.com', 'http://mail.example.com']:
    try:
        response = requests.get(url)

        # If the response was successful, no Exception will be raised
        response.raise_for_status()
    except HTTPError as http_err:
        print(f'HTTP error occurred: {http_err}')  # Python 3.6
    except Exception as err:
        print(f'Other error occurred: {err}')  # Python 3.6
    else:
        response.encoding = 'utf-8' # Optional: requests infers this internally
        soup = BeautifulSoup(response.text, 'lxml')
        links = soup.find_all('a')
        forms = soup.find_all('form')

To install BeautifulSoup use:

pip install beautifulsoup4
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.