646

I'm having trouble parsing HTML elements with "class" attribute using Beautifulsoup. The code looks like this

soup = BeautifulSoup(sdata)
mydivs = soup.findAll('div')
for div in mydivs: 
    if (div["class"] == "stylelistrow"):
        print div

I get an error on the same line "after" the script finishes.

File "./beautifulcoding.py", line 130, in getlanguage
  if (div["class"] == "stylelistrow"):
File "/usr/local/lib/python2.6/dist-packages/BeautifulSoup.py", line 599, in __getitem__
   return self._getAttrMap()[key]
KeyError: 'class'

How do I get rid of this error?

19 Answers 19

1034

You can refine your search to only find those divs with a given class using BS3:

mydivs = soup.find_all("div", {"class": "stylelistrow"})
8
  • @Klaus- what if I want to use findAll instead?
    – user507220
    Apr 21, 2011 at 15:26
  • 68
    This only works for exact matches. <.. class="stylelistrow"> matches but not <.. class="stylelistrow button">.
    – Wernight
    Jul 28, 2014 at 16:07
  • 7
    @pyCthon See answer for @jmunsch, BS now supports class_ which works properly.
    – Wernight
    Oct 6, 2014 at 9:47
  • 3
    _class Will only match the exact class string if you have multiple classes. In that case you can use: soup.select("p.stylelistrow.another") which would match <p class="stylelistrow another yetAnother"> for example
    – smoothware
    Mar 23, 2019 at 13:56
  • 2
    @Wernight Currently using BS4 (4.7.1) and soup.find_all("div", {"class": "stylelistrow"}) works for both exact <.. class="stylelistrow"> and contains <.. class="stylelistrow button"> matches.
    – Girrafish
    Jun 25, 2019 at 14:00
430

From the documentation:

As of Beautiful Soup 4.1.2, you can search by CSS class using the keyword argument class_:

soup.find_all("a", class_="sister")

Which in this case would be:

soup.find_all("div", class_="stylelistrow")

It would also work for:

soup.find_all("div", class_="stylelistrowone stylelistrowtwo")
8
  • 13
    You can use lists too: soup.find_all("a", ["stylelistrowone", "stylelistrow"]) It's safer if you don't have many classes. Jul 7, 2015 at 14:06
  • 7
    This should be the accepted answer, it's both more correct and concise than the alternatives. Mar 13, 2016 at 0:19
  • 3
    Supplement to @NunoAndré's answer for BeautifulSoup 3: soup.findAll("a", {'class':['stylelistrowone', 'stylelistrow']}).
    – Bowen Xu
    Apr 17, 2018 at 15:39
  • 1
    @Timo my guess would be that you can use the dict way when you're searching for an attribute other than class, so maybe something like {'data-item': ['1']} ex. <a data-item="1" />
    – jmunsch
    Jan 31, 2021 at 16:33
  • 2
    One thing to note should be, when you give class_="class_1 class2" it matches "exact string", so even "class_2 class_1" won't match. To search with multiple classes (all required), you should use selectors soup.select('div.class_1.class_2') this matches both "class_1 class_2" and "class_2 class_1".
    – Hrishikesh
    Mar 6, 2021 at 17:46
81

Update: 2016 In the latest version of beautifulsoup, the method 'findAll' has been renamed to 'find_all'. Link to official documentation

List of method names changed

Hence the answer will be

soup.find_all("html_element", class_="your_class_name")
0
52

CSS selectors

single class first match

soup.select_one('.stylelistrow')

list of matches

soup.select('.stylelistrow')

compound class (i.e. AND another class)

soup.select_one('.stylelistrow.otherclassname')
soup.select('.stylelistrow.otherclassname')

Spaces in compound class names e.g. class = stylelistrow otherclassname are replaced with ".". You can continue to add classes.

list of classes (OR - match whichever present)

soup.select_one('.stylelistrow, .otherclassname')
soup.select('.stylelistrow, .otherclassname')

Class attribute whose values contains a string e.g. with "stylelistrow":

starts with "style":

[class^=style]

ends with "row"

[class$=row]

contains "list":

[class*=list]

The ^, $ and * are operators. Read more here: https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors

If you wanted to exclude this class then, with anchor tag as an example, selecting anchor tags without this class:

a:not(.stylelistrow)

You can pass simple, compound and complex css selectors lists inside of :not() pseudo class. See https://facelessuser.github.io/soupsieve/selectors/pseudo-classes/#:not


bs4 4.7.1 +

Specific class whose innerText contains a string

soup.select_one('.stylelistrow:contains("some string")')
soup.select('.stylelistrow:contains("some string")')

N.B.

soupsieve 2.1.0 + Dec'2020 onwards

NEW: In order to avoid conflicts with future CSS specification changes, non-standard pseudo classes will now start with the :-soup- prefix. As a consequence, :contains() will now be known as :-soup-contains(), though for a time the deprecated form of :contains() will still be allowed with a warning that users should migrate over to :-soup-contains().

NEW: Added new non-standard pseudo class :-soup-contains-own() which operates similar to :-soup-contains() except that it only looks at text nodes directly associated with the currently scoped element and not its descendants.

Specific class which has a certain child element e.g. a tag

soup.select_one('.stylelistrow:has(a)')
soup.select('.stylelistrow:has(a)')
1
  • 1
    This worked well for be where I was looking to find by just class and not by element type.
    – Brian C.
    Jun 30, 2021 at 17:52
24

Specific to BeautifulSoup 3:

soup.findAll('div',
             {'class': lambda x: x 
                       and 'stylelistrow' in x.split()
             }
            )

Will find all of these:

<div class="stylelistrow">
<div class="stylelistrow button">
<div class="button stylelistrow">
5
  • Why not re.search('.*stylelistrow.*', x) ?
    – rjurney
    Mar 4, 2015 at 23:55
  • because then stylelistrow2 will match. Better comment is "why not use string.find() instead of re?"
    – FlipMcF
    Mar 5, 2015 at 11:52
  • 4
    lambda x: 'stylelistrow' in x.split() is simple and beautiful
    – fferri
    Jun 24, 2015 at 16:39
  • 1
    And I hate regexes. Thank you! (updating answer) | keeping the 'x and' to test for None
    – FlipMcF
    Jul 17, 2015 at 22:03
  • "keeping the 'x and' to test for None" I have found this for a long time Jul 31, 2022 at 4:12
19

A straight forward way would be :

soup = BeautifulSoup(sdata)
for each_div in soup.findAll('div',{'class':'stylelist'}):
    print each_div

Make sure you take of the casing of findAll, its not findall

1
  • 6
    This only works for exact matches. <.. class="stylelistrow"> matches but not <.. class="stylelistrow button">.
    – Wernight
    Jul 28, 2014 at 16:07
16

Use class_= If you want to find element(s) without stating the HTML tag.

For single element:

soup.find(class_='my-class-name')

For multiple elements:

soup.find_all(class_='my-class-name')
15

How to find elements by class

I'm having trouble parsing html elements with "class" attribute using Beautifulsoup.

You can easily find by one class, but if you want to find by the intersection of two classes, it's a little more difficult,

From the documentation (emphasis added):

If you want to search for tags that match two or more CSS classes, you should use a CSS selector:

css_soup.select("p.strikeout.body")
# [<p class="body strikeout"></p>]

To be clear, this selects only the p tags that are both strikeout and body class.

To find for the intersection of any in a set of classes (not the intersection, but the union), you can give a list to the class_ keyword argument (as of 4.1.2):

soup = BeautifulSoup(sdata)
class_list = ["stylelistrow"] # can add any other classes to this list.
# will find any divs with any names in class_list:
mydivs = soup.find_all('div', class_=class_list) 

Also note that findAll has been renamed from the camelCase to the more Pythonic find_all.

11

As of BeautifulSoup 4+ ,

If you have a single class name , you can just pass the class name as parameter like :

mydivs = soup.find_all('div', 'class_name')

Or if you have more than one class names , just pass the list of class names as parameter like :

mydivs = soup.find_all('div', ['class1', 'class2'])
4

the following worked for me

a_tag = soup.find_all("div",class_='full tabpublist')
4

single

soup.find("form",{"class":"c-login__form"})

multiple

res=soup.find_all("input")
for each in res:
    print(each)
3

This works for me to access the class attribute (on beautifulsoup 4, contrary to what the documentation says). The KeyError comes a list being returned not a dictionary.

for hit in soup.findAll(name='span'):
    print hit.contents[1]['class']
3

Other answers did not work for me.

In other answers the findAll is being used on the soup object itself, but I needed a way to do a find by class name on objects inside a specific element extracted from the object I obtained after doing findAll.

If you are trying to do a search inside nested HTML elements to get objects by class name, try below -

# parse html
page_soup = soup(web_page.read(), "html.parser")

# filter out items matching class name
all_songs = page_soup.findAll("li", "song_item")

# traverse through all_songs
for song in all_songs:

    # get text out of span element matching class 'song_name'
    # doing a 'find' by class name within a specific song element taken out of 'all_songs' collection
    song.find("span", "song_name").text

Points to note:

  1. I'm not explicitly defining the search to be on 'class' attribute findAll("li", {"class": "song_item"}), since it's the only attribute I'm searching on and it will by default search for class attribute if you don't exclusively tell which attribute you want to find on.

  2. When you do a findAll or find, the resulting object is of class bs4.element.ResultSet which is a subclass of list. You can utilize all methods of ResultSet, inside any number of nested elements (as long as they are of type ResultSet) to do a find or find all.

  3. My BS4 version - 4.9.1, Python version - 3.8.1

3

Concerning @Wernight's comment on the top answer about partial matching...

You can partially match:

  • <div class="stylelistrow"> and
  • <div class="stylelistrow button">

with gazpacho:

from gazpacho import Soup

my_divs = soup.find("div", {"class": "stylelistrow"}, partial=True)

Both will be captured and returned as a list of Soup objects.

2

Alternatively we can use lxml, it support xpath and very fast!

from lxml import html, etree 

attr = html.fromstring(html_text)#passing the raw html
handles = attr.xpath('//div[@class="stylelistrow"]')#xpath exresssion to find that specific class

for each in handles:
    print(etree.tostring(each))#printing the html as string
1

Try to check if the div has a class attribute first, like this:

soup = BeautifulSoup(sdata)
mydivs = soup.findAll('div')
for div in mydivs:
    if "class" in div:
        if (div["class"]=="stylelistrow"):
            print div
2
  • 1
    That doesn't work. I guess your approach was right, but 4th line doesn't work as intended.
    – Neo
    Feb 18, 2011 at 12:08
  • 1
    Ah I thought div worked like a dictionary, I'm not really familiar with Beautiful Soup so it was just a guess.
    – Mew
    Feb 18, 2011 at 12:11
1

This worked for me:

for div in mydivs:
    try:
        clazz = div["class"]
    except KeyError:
        clazz = ""
    if (clazz == "stylelistrow"):
        print div
1

This should work:

soup = BeautifulSoup(sdata)
mydivs = soup.findAll('div')
for div in mydivs: 
    if (div.find(class_ == "stylelistrow"):
        print div
1

The following should work

soup.find('span', attrs={'class':'totalcount'})

replace 'totalcount' with your class name and 'span' with tag you are looking for. Also, if your class contains multiple names with space, just choose one and use.

P.S. This finds the first element with given criteria. If you want to find all elements then replace 'find' with 'find_all'.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.