3

I'm kind of just getting started with Python, and I'm trying to match the top 250 movies on IMDB with this malfunctioning code:

import urllib2
import re

def main():
    response = urllib2.urlopen('http://www.imdb.com/chart/top')
    html = response.read()
    entries = re.findall("/title/.*</font>", html) #Wrong regex
    print entries

if __name__ == "__main__":
    main()

My rationale is that I want to match everything between /title/ and </font>, hence the .* in between, but obviously this isn't the right way to go here since it simply matches the entire list instead of each individual entry. I'm pretty confused by regex tutorials I've been perusing online.... Help?

1

5 Answers 5

5

So, trying to parse HTML using regex is a bad practice to handle these kind of things html parsers are built. There are many option available in python like Beautiful Soup, lxml etc.

I am going to show how to use lxml with XPath expressions to fetch all the top 250 titles

import lxml
from lxml import etree
import urllib2

response = urllib2.urlopen('http://www.imdb.com/chart/top')
html = response.read()
imdb = etree.HTML(html)
titles = imdb.xpath('//div[@id="main"]/table//tr//a/text()')

if you do print titles[0] it will give 'The Shawshank Redemption' as output. For, XPath use firefox's firebug extension or install firepath

1
  • 1
    It can be simplified to titles = imdb.xpath('//div[@id="main"]/table//a/text()')
    – reclosedev
    Jan 6, 2012 at 9:43
2

try this

def main(s):
    response = urllib2.urlopen('http://www.imdb.com/chart/top')
    html = response.read()
    entries = re.findall("<a.*?/title/(.*?)/\">(.*?)</a>", html) #Wrong regex
    return entries

it uses groups for the imdb id and the title. entries will be a list of tuples

1
  • 1
    Yeah, that definitely works perfectly. Trying to parse Unicode characters like so using the codecs package: html = codecs.open(response.read(), "r", "utf-8"), though doesn't entirely seem to work..
    – benjammin
    Jan 6, 2012 at 8:58
1

You shouldn't use regexes for html parsing. You should use specialized html parsers. Have a look at: RegEx match open tags except XHTML self-contained tags

2
  • 2
    He's trying to parse a specific webpage, not generalized HTML. Regexes will work fine for his purposes.
    – Taymon
    Jan 6, 2012 at 8:42
  • 1
    @Taymon: even if you are trying to parse a specific webpage you should use html parsers they are built to do these things
    – RanRag
    Jan 6, 2012 at 8:44
1

It can be done simple with lxml and XPath:

import lxml.html

doc = lxml.html.parse('http://www.imdb.com/chart/top')
titles  = doc.xpath('//div[@id="main"]/table//a/text()')

print u'\n'.join(titles)
0
0

This is a long answer; I'm hoping it will both get you what you need and help to elucidate the mystery of regexes.

The way to go about this is to select an entire individual entry, then replace the parts that you want to generalize with patterns that will match them.

For example, the first entry in the page you mentioned looks like this.

<tr bgcolor="#e5e5e5" valign="top"><td align="right"><font face="Arial, Helvetica, sans-serif" size="-1"><b>1.</b></font></td><td align="center"><font face="Arial, Helvetica, sans-serif" size="-1">9.2</font></td><td><font face="Arial, Helvetica, sans-serif" size="-1"><a href="/title/tt0111161/">The Shawshank Redemption</a> (1994)</font></td><td align="right"><font face="Arial, Helvetica, sans-serif" size="-1">689,815</font></td></tr>

All the entries look like this, but there are some things that are different from entry to entry. I'll go through them one by one.

The background color, as you can see on the page, alternates between gray (#e5e5e5) and white (#ffffff). You want to be able to match either of those. To do so, you can use the | operator, which allows you to specify multiple patterns, any of which will be accepted. That part of the entry will be replaced with:

bgcolor="#(?:e5e5e5|ffffff)"

The parentheses make it so that the | operator divides only that one part of the expression, as opposed to all of it. The ?: after the opening parenthesis prevents this parenthetical expression from being captured as a group, which we don't want because we don't care what color a particular movie was. More on this later.

The next thing that varies from entry to entry is what number entry it is. We'll replace that part of the entry with this:

(\d{1,3})\.

Whenever you see a backslash in a regex, it does something special; what it does depends on the following character. \d matches any decimal digit. The {1,3} means that it expects between one and three of them, since all the entry numbers are between one and three digits.

The other major use of backslashes is to literally represent characters that do something special. The dot is such a character; it matches any character. But that's not what we want; we want a literal .. To get that, we use \., which matches .. The characters that you need to do this with are called metacharacters, and they are these: .^$*+?|{}[]()\

Finally, we added parentheses around the numbers. In addition to subdividing a regular expression, parentheses also form a group; that is, they indicate that you care about this part of the regex. When you use re.findall, only the parts of the matches that are in groups are saved; everything else is discarded. (The ?: that was mentioned above is for when you don't care about part of the regex, but need to use parentheses anyway to subdivide it.) I'm assuming that all information in the table, including the entry number, is important information, so I've enclosed the entry number (but not the following period, which we don't care about) in parentheses. If you don't care about the entry number, you can get rid of the parentheses.

After that comes the movie's score, which we handle similarly:

(\d\.\d)

Next up are the links to the movies. The only thing that changes here are the numbers in the links, which are always seven digits. So we'll replace that part of the entry with:

<a href="/title/tt(\d{7})/">

Now we have to deal with the movie title. There are a few different ways to go about this; the most important thing is to avoid ambiguity. You have to make clear that the < following the title is not part of it (which is why your first attempt didn't do what you wanted). One way to do it would be to declare that a movie title can only contain letters, digits, spaces, and certain punctuation marks. To do that, you could say this:

([A-Za-z0-9 ,.:'-]+)

Square brackets mean that it will match any of the characters contained inside. These can be literal characters (as in the punctuation marks and spaces) or ranges of characters (for instance, A-Z matches all uppercase letters). Ranges are denoted by a hyphen between the beginning and ending characters. The hyphen at the end is a literal hyphen; it's at the end so as not to confuse the regex parser into thinking it's part of a range. Note that most metacharacters don't have to be escaped with a backslash inside a character group.

Meanwhile, the + means to repeat whatever precedes it one or more times. Put all together, each movie title consists of one or more characters from the set I've chosen. Since < is not in that set, there's no question that the title ends there.

This seems like a good pattern for the title, but there's a problem with it. Two of the movie titles on the page (WALL·E and ) contain exotic characters that the pattern won't match. Rather than trying to come up with every possible character that could ever appear in a movie title, I'll use a much simpler pattern:

([^<]+)

The ^ at the beginning of the character set means that instead of matching any character in the set, it should match every character not in the set (excluding the ^ itself). This way, a movie title can contain any character (so we don't have to guess which ones are valid), with the sole exception of < (because, as explained, we can't allow that ambiguity). Fortunately, even if there were a movie with < in the title, the rules of HTML state that it would have to be escaped as &lt;, so it would still match. This set isn't ideal because it's not very specific, but it's easy and there's not much of a better choice.

Now for the movie's year:

\((\d{4})\)

This is slightly confusing; the outer parentheses are escaped with backslashes because they represent literal parentheses, but the inner ones aren't because they're needed to form a group (since we care what year a movie was).

Finally, the number of votes. This is a comma-grouped positive number, so we'll match it like so:

(\d{1,3}(?:,\d{3})*)

Again, this one's complicated. The gist of it is that the number will consist of one or more groups separated by commas. The first group can have between one and three digits; the rest will all have exactly three. The * is similar to +, but it matches zero or more repetitions instead of one or more. The inner set of parentheses is needed so that the * matches everything inside it, instead of just the preceding character. It has a ?: because we care about the number as a whole instead of individual groups in it.

Now it's time to put it all together. In doing so, we have to make one final consideration: making sure the regex parser sees all those backslashes, so that they do what they're supposed to. The reason we have to worry about this is because Python itself uses backslashes as an escape character, and if it processes them before the regex parser does, it won't work. To prevent this, regexes are almost always written using raw strings, that is, strings with an r placed before the opening quote. This tells Python to leave the backslashes alone. Also, since the regex has lots of double quotes in it, we'll want to put it in a single-quoted string.

With that said, the line of code as it appears in your program will look like this:

entries = re.findall(r'<tr bgcolor="#(?:e5e5e5|ffffff)" valign="top"><td align="right"><font face="Arial, Helvetica, sans-serif" size="-1"><b>(\d{1,3})\.</b></font></td><td align="center"><font face="Arial, Helvetica, sans-serif" size="-1">(\d\.\d)</font></td><td><font face="Arial, Helvetica, sans-serif" size="-1"><a href="/title/tt(\d{7})/">([^<]+)</a> \((\d{4})\)</font></td><td align="right"><font face="Arial, Helvetica, sans-serif" size="-1">(\d{1,3}(?:,\d{3})*)</font></td></tr>', html)

This will get you a list of matches. Each match will be a tuple consisting of all the groups (number, score, link number, title, year, votes) in order. Note that all of these, even the ones representing numeric quantities, will be strings.

If you want to learn more about regexes, I recommend this site. Also worth reading are the Python docs for the re module.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.