55

I'm trying to extract some text using BeautifulSoup. I'm using get_text() function for this purpose.

My problem is that the text contains </br> tags and I need to convert them to end lines. how can I do this?

1
  • I know this is 10 years old but did you mean <br/> and not </br>? I'd simply edit it but since there is a possibility that you actually mean broken </br> tags (and were misunderstood by the answerers), I though I'd ask.
    – CherryDT
    Dec 22, 2022 at 8:42

7 Answers 7

92

You can do this using the BeautifulSoup object itself, or any element of it:

for br in soup.find_all("br"):
    br.replace_with("\n")
2
  • 3
    the benefit of this answer is, that you can call soup.text afterwards to remove other html tags, whereas the currently accepted answer doesn't provide that possibility.
    – the
    Apr 11, 2017 at 15:21
  • 9
    Watch out for this, you may end up losing some contents unintentionally. You may need to do something like br.replace_with("\n" + br.text). This tag is evil... Feb 21, 2018 at 9:52
70

As official doc says:

You can specify a string to be used to join the bits of text together: soup.get_text("\n")

2
  • 1
    Seems that "bits of text" are words, not lines, so this would add a newline between each 2 words.
    – Sasha
    Apr 2, 2018 at 11:56
  • 3
    @Sasha I'm not sure what you mean by that -- I believe "bits of text" refers to text separated by tags. I am certainly not getting a newline between each pair of words when I run it, as you suggest.
    – Y Davis
    Jul 16, 2019 at 17:54
9

Also you can use ‍‍‍get_text(separator = '\n', strip = True) :

from bs4 import BeautifulSoup
bs=BeautifulSoup('<td>some text<br>some more text</td>','html.parser')
text=bs.get_text(separator = '\n', strip = True)
print(text)
 >> 
some text
some more text

it works for me.

5

A regex should do the trick.

import re
s = re.sub('<br\s*?>', '\n', yourTextHere)

Hope this helps!

1
  • This will fail to convert self-closing tags
    – CherryDT
    Dec 22, 2022 at 8:39
5

Adding to Ian's and dividebyzero's post/comments you can do this to efficiently filter/replace many tags in one go:

for elem in soup.find_all(["a", "p", "div", "h3", "br"]):
    elem.replace_with(elem.text + "\n\n")
1

Instead of replacing the tags with \n, it may be better to just add a \n to the end of all of the tags that matter.

To steal the list from @petezurich:

for elem in soup.find_all(["a", "p", "div", "h3", "br"]):
    elem.append('\n')
1

If you call element.text you'll get the text without br tags. Maybe you need define your own custom method for this purpose:

     def clean_text(elem):
        text = ''
        for e in elem.descendants:
            if isinstance(e, str):
                text += e.strip()
            elif e.name == 'br' or e.name == 'p':
                text += '\n'
        return text

    # get page content
    soup = BeautifulSoup(request_response.text, 'html.parser')
    # get your target element
    description_div = soup.select_one('.description-class')
    # clean the data
    print(clean_text(description_div))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.