1

I am trying to write a script to scrape a website, and am using this one (http://www.theericwang.com/scripts/eBayRead.py).

I however want to use it to crawl sites other than ebay, and to customize to my needs.

I am fairly new to python and have limited re experience.

I am unsure of what this line achieves.

for url, title in re.findall(r'href="([^"]+).*class="vip" title=\'([^\']+)', lines):

Could someone please give me some pointers?

Is there anything else I need to consider if I port this for other sites?

2

3 Answers 3

3

In general, parsing HTML is best done with a library such as BeautifulSoup, which takes care of virtually all of the heavy lifting for you, leaving you with more intuitive code. Also, read @Tadeck's link below - regex and HTML shouldn't be mixed if it can be avoided (to put it lightly).

As for your question, that line uses something called 'regular expression' to find matching patterns in a text (in this case, HTML). re.findall() is a method that returns a list, so if we focus on just that:

re.findall(r'href="([^"]+).*class="vip" title=\'([^\']+)', lines):

r indicates that the following will be interpreted 'raw', meaning that characters like backslashes, etc., will be interpreted literally.

href="([^"]+)

The parentheses indicate a group (what we care about in the match), and the [^"]+ means 'match anything that isn't a quote'. As you can probably guess, this group will return the URL of the link.

.*class="vip"

The .* matches anything (well, almost anything) 0 or more times (which here could include other tags, the closing quote of the link, whitespace, etc.). Nothing special with class="vip" - it just needs to appear.

title=\'([^\']+)', lines):

Here you see an escaped quote and then another group as we saw above. This time, we are capturing anything between the two apostrophes after the title tag.

The end result of this is you are iterating through a list of all matches, and those matches are going to look something like (my_matched_link, my_matched_title), which are passed into for url, title, after which further processing is done.

3

I am not sure if this would answer your question. But you can consider scrapy: http://scrapy.org for crawling various websites. It is a nice infrastructure which provides a lot of flexibility and is easy to customize to some specific needs.

2

Regular expressions are bad for parsing HTML

The above is the main idea I would like to communicate to you. For why, see this question: RegEx match open tags except XHTML self-contained tags.

In short, HTML can change as a text (eg. new attribute can be added, order of attributes can be changed, or some other changes may be introduced), but it will result in the exact same HTML as interpreted by web browsers, while completely breaking your script.

The HTML should be parsed using specialized HTML parsers or web scrapers. They know the difference, when it becomes significant.

What to use for scraping?

There are multiple solutions, but one of the most notable ones is: ScraPy. Try it, you may start to love it.

3
  • Ha, was just about to paste that link into my answer :) Nov 15, 2012 at 5:16
  • @RocketDonkey: So do it, I do not hold copyright rights for this link ;) Actually, the question is the 9th question on SO with the biggest number of upvotes, so hopefully it is widely known, and decreases number of questions on "why regexp does not work with HTML" or "should I use regexp to parse HTML" :)
    – Tadeck
    Nov 15, 2012 at 5:22
  • Ha, good point - I think that answer alone has caused me to have a physical reaction whenever I think/hear about using regex with HTML. Also makes me laugh every time :) Nov 15, 2012 at 5:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.