133

How would I, using BeautifulSoup, search for tags containing ONLY the attributes I search for?

For example, I want to find all <td valign="top"> tags.

The following code: raw_card_data = soup.fetch('td', {'valign':re.compile('top')})

gets all of the data I want, but also grabs any <td> tag that has the attribute valign:top

I also tried: raw_card_data = soup.findAll(re.compile('<td valign="top">')) and this returns nothing (probably because of bad regex)

I was wondering if there was a way in BeautifulSoup to say "Find <td> tags whose only attribute is valign:top"

UPDATE FOr example, if an HTML document contained the following <td> tags:

<td valign="top">.....</td><br />
<td width="580" valign="top">.......</td><br />
<td>.....</td><br />

I would want only the first <td> tag (<td width="580" valign="top">) to return

9 Answers 9

149

As explained on the BeautifulSoup documentation

You may use this :

soup = BeautifulSoup(html)
results = soup.findAll("td", {"valign" : "top"})

EDIT :

To return tags that have only the valign="top" attribute, you can check for the length of the tag attrs property :

from BeautifulSoup import BeautifulSoup

html = '<td valign="top">.....</td>\
        <td width="580" valign="top">.......</td>\
        <td>.....</td>'

soup = BeautifulSoup(html)
results = soup.findAll("td", {"valign" : "top"})

for result in results :
    if len(result.attrs) == 1 :
        print result

That returns :

<td valign="top">.....</td>
2
  • As per my comment to julio.alegria, This will find all <tr> tags with the attribute valign="top", including ones with other attributes (<td width="580" valign="top"> is also returned in this search) I'm looking for a method to find <tr> tags whose only attribute is valign="top"
    – Snaxib
    Commented Jan 19, 2012 at 22:37
  • So, you can check len(tag.attrs). If len(tag.attrs) > 1, ignore the tag (I have edited my post)
    – Loïc G.
    Commented Jan 19, 2012 at 22:52
67

You can use lambda functions in findAll as explained in documentation. So that in your case to search for td tag with only valign = "top" use following:

td_tag_list = soup.findAll(
                lambda tag:tag.name == "td" and
                len(tag.attrs) == 1 and
                tag["valign"] == "top")
3
  • 6
    best answer as it uses the full power of BS
    – Rafael T
    Commented Oct 30, 2014 at 4:21
  • 2
    Great answer because its gives you result in very optimized way.
    – CrazyGeek
    Commented Jan 7, 2015 at 12:24
  • Risks a KeyError if the tag isn't actually valign. Commented Dec 14, 2023 at 7:33
61

if you want to only search with attribute name with any value

from bs4 import BeautifulSoup
import re

soup= BeautifulSoup(html.text,'lxml')
results = soup.findAll("td", {"valign" : re.compile(r".*")})

as per Steve Lorimer better to pass True instead of regex

results = soup.findAll("td", {"valign" : True})
2
  • 3
    You're missing a parenthesis after r".*", resulting in this not compiling.
    – Jack Cole
    Commented Dec 20, 2017 at 19:02
  • 24
    No need for a regular expression, just pass True: results = soup.findAll("td", {"valign" : True}) Commented Feb 25, 2019 at 20:25
20

The easiest way to do this is with the new CSS style select method:

soup = BeautifulSoup(html)
results = soup.select('td[valign="top"]')
2
  • 1
    What if I don't know the value beforehand, and just want to find tag containing the valign attribute? Commented Dec 4, 2020 at 1:18
  • 2
    @MasayoMusic You can do soup.select('td[valign]') to select all <td> having 'valign' attribute. You can also omit the tag name to select all elements with 'valign' attribute
    – Elben
    Commented Dec 16, 2020 at 12:00
6

find using an attribute in any tag

<th class="team" data-sort="team">Team</th>    
soup.find_all(attrs={"class": "team"}) 

<th data-sort="team">Team</th>  
soup.find_all(attrs={"data-sort": "team"}) 
 
6

If you are looking to pull all tags where a particular attribute is present at all, you can use the same code as the accepted answer, but instead of specifying a value for the tag, just put True.

soup = BeautifulSoup(html)
results = soup.findAll("td", {"valign" : True})

This will return all td tags that have valign attributes. This is useful if your project involves pulling info from a tag like div that is used all over, but can handle very specific attributes that you might be looking for.

5

Just pass it as an argument of findAll:

>>> from BeautifulSoup import BeautifulSoup
>>> soup = BeautifulSoup("""
... <html>
... <head><title>My Title!</title></head>
... <body><table>
... <tr><td>First!</td>
... <td valign="top">Second!</td></tr>
... </table></body><html>
... """)
>>>
>>> soup.findAll('td')
[<td>First!</td>, <td valign="top">Second!</td>]
>>>
>>> soup.findAll('td', valign='top')
[<td valign="top">Second!</td>]
1
  • 2
    What if there are tags like so: <td width="580" valign="top">? I don't want to grab those, just tags whose only attribute is valign="top"
    – Snaxib
    Commented Jan 19, 2012 at 22:27
4

Adding a combination of Chris Redford's and Amr's answer, you can also search for an attribute name with any value with the select command:

from bs4 import BeautifulSoup as Soup
html = '<td valign="top">.....</td>\
    <td width="580" valign="top">.......</td>\
    <td>.....</td>'
soup = Soup(html, 'lxml')
results = soup.select('td[valign]')
2
  • I have tried the same way but this is not working, is there any workaround? Commented Feb 21, 2019 at 12:07
  • 1
    @PhaneendraCharyuluKanduri Sorry, there was a bad coding error in the code. Now copy and paste should work!
    – Adrian
    Commented Feb 22, 2019 at 15:06
1

If you want to print the name of all the tags in different lines that have specific attribute , for example printing all the tags with id attribute irrespective of the value :

from bs4 import BeautifulSoup ;
from bs4 import element ;
html = '!DOCTYPE html><html><head><title>Navigate Parse Tree</title></head>\
<body><h1>This is your Assignment</h1><a href = "https://www.google.com">This is a link that will take you to Google</a>\
<ul><li><p> This question is given to test your knowledge of <b>Web Scraping</b></p>\
<p>Web scraping is a term used to describe the use of a program or algorithm to extract and process large amounts of data from the web.</p></li>\
<li id = "li2">This is an li tag given to you for scraping</li>\
<li>This li tag gives you the various ways to get data from a website\
<ol><li class = "list_or">Using API of the website</li><li>Scrape data using BeautifulSoup</li><li>Scrape data using Selenium</li>\
<li>Scrape data using Scrapy</li></ol></li>\
<li class = "list_or"><a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">\
Clicking on this takes you to the documentation of BeautifulSoup</a>\
<a href="https://selenium-python.readthedocs.io/" id="anchor">Clicking on this takes you to the documentation of Selenium</a>\
</li></ul></body></html>'

data = BeautifulSoup(html, 'html.parser');
for i in data.descendants :
     if type(i) == element.Tag:
        if i.attrs != {} and 'id' in i.attrs:
           print(i.name)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.