129

How would I, using BeautifulSoup, search for tags containing ONLY the attributes I search for?

For example, I want to find all <td valign="top"> tags.

The following code: raw_card_data = soup.fetch('td', {'valign':re.compile('top')})

gets all of the data I want, but also grabs any <td> tag that has the attribute valign:top

I also tried: raw_card_data = soup.findAll(re.compile('<td valign="top">')) and this returns nothing (probably because of bad regex)

I was wondering if there was a way in BeautifulSoup to say "Find <td> tags whose only attribute is valign:top"

UPDATE FOr example, if an HTML document contained the following <td> tags:

<td valign="top">.....</td><br />
<td width="580" valign="top">.......</td><br />
<td>.....</td><br />

I would want only the first <td> tag (<td width="580" valign="top">) to return

9 Answers 9

145

As explained on the BeautifulSoup documentation

You may use this :

soup = BeautifulSoup(html)
results = soup.findAll("td", {"valign" : "top"})

EDIT :

To return tags that have only the valign="top" attribute, you can check for the length of the tag attrs property :

from BeautifulSoup import BeautifulSoup

html = '<td valign="top">.....</td>\
        <td width="580" valign="top">.......</td>\
        <td>.....</td>'

soup = BeautifulSoup(html)
results = soup.findAll("td", {"valign" : "top"})

for result in results :
    if len(result.attrs) == 1 :
        print result

That returns :

<td valign="top">.....</td>
2
  • As per my comment to julio.alegria, This will find all <tr> tags with the attribute valign="top", including ones with other attributes (<td width="580" valign="top"> is also returned in this search) I'm looking for a method to find <tr> tags whose only attribute is valign="top"
    – Snaxib
    Jan 19, 2012 at 22:37
  • So, you can check len(tag.attrs). If len(tag.attrs) > 1, ignore the tag (I have edited my post)
    – Loïc G.
    Jan 19, 2012 at 22:52
65

You can use lambda functions in findAll as explained in documentation. So that in your case to search for td tag with only valign = "top" use following:

td_tag_list = soup.findAll(
                lambda tag:tag.name == "td" and
                len(tag.attrs) == 1 and
                tag["valign"] == "top")
2
  • 6
    best answer as it uses the full power of BS
    – Rafael T
    Oct 30, 2014 at 4:21
  • 2
    Great answer because its gives you result in very optimized way.
    – CrazyGeek
    Jan 7, 2015 at 12:24
56

if you want to only search with attribute name with any value

from bs4 import BeautifulSoup
import re

soup= BeautifulSoup(html.text,'lxml')
results = soup.findAll("td", {"valign" : re.compile(r".*")})

as per Steve Lorimer better to pass True instead of regex

results = soup.findAll("td", {"valign" : True})
2
  • 3
    You're missing a parenthesis after r".*", resulting in this not compiling.
    – Jack Cole
    Dec 20, 2017 at 19:02
  • 23
    No need for a regular expression, just pass True: results = soup.findAll("td", {"valign" : True}) Feb 25, 2019 at 20:25
20

The easiest way to do this is with the new CSS style select method:

soup = BeautifulSoup(html)
results = soup.select('td[valign="top"]')
2
  • 1
    What if I don't know the value beforehand, and just want to find tag containing the valign attribute? Dec 4, 2020 at 1:18
  • 2
    @MasayoMusic You can do soup.select('td[valign]') to select all <td> having 'valign' attribute. You can also omit the tag name to select all elements with 'valign' attribute
    – Elben
    Dec 16, 2020 at 12:00
5

Just pass it as an argument of findAll:

>>> from BeautifulSoup import BeautifulSoup
>>> soup = BeautifulSoup("""
... <html>
... <head><title>My Title!</title></head>
... <body><table>
... <tr><td>First!</td>
... <td valign="top">Second!</td></tr>
... </table></body><html>
... """)
>>>
>>> soup.findAll('td')
[<td>First!</td>, <td valign="top">Second!</td>]
>>>
>>> soup.findAll('td', valign='top')
[<td valign="top">Second!</td>]
1
  • 2
    What if there are tags like so: <td width="580" valign="top">? I don't want to grab those, just tags whose only attribute is valign="top"
    – Snaxib
    Jan 19, 2012 at 22:27
4

Adding a combination of Chris Redford's and Amr's answer, you can also search for an attribute name with any value with the select command:

from bs4 import BeautifulSoup as Soup
html = '<td valign="top">.....</td>\
    <td width="580" valign="top">.......</td>\
    <td>.....</td>'
soup = Soup(html, 'lxml')
results = soup.select('td[valign]')
2
  • I have tried the same way but this is not working, is there any workaround? Feb 21, 2019 at 12:07
  • 1
    @PhaneendraCharyuluKanduri Sorry, there was a bad coding error in the code. Now copy and paste should work!
    – Adrian
    Feb 22, 2019 at 15:06
4

find using an attribute in any tag

<th class="team" data-sort="team">Team</th>    
soup.find_all(attrs={"class": "team"}) 

<th data-sort="team">Team</th>  
soup.find_all(attrs={"data-sort": "team"}) 
 
4

If you are looking to pull all tags where a particular attribute is present at all, you can use the same code as the accepted answer, but instead of specifying a value for the tag, just put True.

soup = BeautifulSoup(html)
results = soup.findAll("td", {"valign" : True})

This will return all td tags that have valign attributes. This is useful if your project involves pulling info from a tag like div that is used all over, but can handle very specific attributes that you might be looking for.

0

If you want to print the name of all the tags in different lines that have specific attribute , for example printing all the tags with id attribute irrespective of the value :

from bs4 import BeautifulSoup ;
from bs4 import element ;
html = '!DOCTYPE html><html><head><title>Navigate Parse Tree</title></head>\
<body><h1>This is your Assignment</h1><a href = "https://www.google.com">This is a link that will take you to Google</a>\
<ul><li><p> This question is given to test your knowledge of <b>Web Scraping</b></p>\
<p>Web scraping is a term used to describe the use of a program or algorithm to extract and process large amounts of data from the web.</p></li>\
<li id = "li2">This is an li tag given to you for scraping</li>\
<li>This li tag gives you the various ways to get data from a website\
<ol><li class = "list_or">Using API of the website</li><li>Scrape data using BeautifulSoup</li><li>Scrape data using Selenium</li>\
<li>Scrape data using Scrapy</li></ol></li>\
<li class = "list_or"><a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">\
Clicking on this takes you to the documentation of BeautifulSoup</a>\
<a href="https://selenium-python.readthedocs.io/" id="anchor">Clicking on this takes you to the documentation of Selenium</a>\
</li></ul></body></html>'

data = BeautifulSoup(html, 'html.parser');
for i in data.descendants :
     if type(i) == element.Tag:
        if i.attrs != {} and 'id' in i.attrs:
           print(i.name)

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.