25

I'm working on a web scraping project and have ran into problems with speed. To try to fix it, I want to use lxml instead of html.parser as BeautifulSoup's parser. I've been able to do this:

soup = bs4.BeautifulSoup(html, 'lxml')

but I don't want to have to repeatedly type 'lxml' every time I call BeautifulSoup. Is there a way I can set which parser to use once at the beginning of my program?

3
  • 1
    lxml is the default in bs4, assuming you have lxml installed. So unless you happen to be working with BeautifulSoup3...
    – roippi
    Jan 6, 2015 at 0:57
  • I am using bs4, but I didn't know how to check which parser I was currently using. Thank you! Jan 6, 2015 at 1:00
  • Related to stackoverflow.com/questions/33511544 containing other details.
    – bufh
    Jun 30, 2020 at 13:57

2 Answers 2

24

According to the Specifying the parser to use documentation page:

The first argument to the BeautifulSoup constructor is a string or an open filehandle–the markup you want parsed. The second argument is how you’d like the markup parsed.

If you don’t specify anything, you’ll get the best HTML parser that’s installed. Beautiful Soup ranks lxml’s parser as being the best, then html5lib’s, then Python’s built-in parser.

In other words, just installing lxml in the same python environment makes it a default parser.

Though note, that explicitly stating a parser is considered a best-practice approach. There are differences between parsers that can result into subtle errors which would be difficult to debug if you are letting BeautifulSoup choose the best parser by itself. You would also have to remember that you need to have lxml installed. And, if you would not have it installed, you would not even notice it - BeautifulSoup would just get the next available parser without throwing any errors.

If you still don't want to specify the parser explicitly, at least make a note for future yourself or others who would use the code you've written in the project's README/documentation, and list lxml in your project requirements alongside with beautifulsoup4.

Besides: "Explicit is better than implicit."

1
  • Note, with bs4 version 4.5.1, when specifuing 'lxml' parser and not having it installed bs4 does error out: bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?
    – glexey
    Dec 15, 2016 at 23:10
12

Obviously take a look at the accepted answer first. It is pretty good, and as for this technicality:

but I don't want to have to repeatedly type 'lxml' every time I call BeautifulSoup. Is there a way I can set which parser to use once at the beginning of my program?

If I understood your question correctly, I can think of two approaches that will save you some keystrokes: - Define a wrapper function, or - Create a partial function.

# V1 - define a wrapper function - most straight-forward.
import bs4

def bs_parse(html):
    return bs4.BeautifulSoup(html, 'lxml')
# ...
html = ...
bs_parse(html)

Or if you feel like showing off ...

import bs4
from functools import partial
bs_parse = partial(bs4.BeautifulSoup, features='lxml')
# ...
html = ...
bs_parse(html)
3
  • Can you add explanation to how partial works? Are there any advantages to using it over a wrapper function?
    – rer
    Aug 1, 2017 at 22:52
  • 1
    @r3robertson There is good documentation for partial functions here: docs.python.org/2/library/functools.html#functools.partial It looks to me like partial is both slower and more complicated under the hood as compared to a wrapper, but once it has been implemented, it is rather easy to use. Partial functions are clean for a lack of a better word from mathematical point of view. Other languages have this and it is considered a good use of functional programming by some, but you do pay a price in terms of speed and an extra import. I still use partial functions because they are fun.
    – Leonid
    Aug 2, 2017 at 3:49
  • 1
    @r3robertson By the way, partial functions do not have to create extra overhead when using certain compiled languages that make performance a priority, but Python is not one of those languages. C++ is however vittorioromeo.info/index/blog/cpp17_curry.html
    – Leonid
    Nov 5, 2020 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.