68

I'd like to take an HTML table and parse through it to get a list of dictionaries. Each list element would be a dictionary corresponding to a row in the table.

If, for example, I had an HTML table with three columns (marked by header tags), "Event", "Start Date", and "End Date" and that table had 5 entries, I would like to parse through that table to get back a list of length 5 where each element is a dictionary with keys "Event", "Start Date", and "End Date".

Thanks for the help!

4 Answers 4

86

You should use some HTML parsing library like lxml:

from lxml import etree
s = """<table>
  <tr><th>Event</th><th>Start Date</th><th>End Date</th></tr>
  <tr><td>a</td><td>b</td><td>c</td></tr>
  <tr><td>d</td><td>e</td><td>f</td></tr>
  <tr><td>g</td><td>h</td><td>i</td></tr>
</table>
"""
table = etree.HTML(s).find("body/table")
rows = iter(table)
headers = [col.text for col in next(rows)]
for row in rows:
    values = [col.text for col in row]
    print dict(zip(headers, values))

prints

{'End Date': 'c', 'Start Date': 'b', 'Event': 'a'}
{'End Date': 'f', 'Start Date': 'e', 'Event': 'd'}
{'End Date': 'i', 'Start Date': 'h', 'Event': 'g'}
8
  • My table has a varying number of rows. How can I make it work if this is the case? Thanks for the response, btw.
    – Andrew
    Jun 12, 2011 at 23:09
  • 1
    @Andrew: The above code works for any number of rows and any number of columns, as long as every row has the same number of columns. Jun 12, 2011 at 23:44
  • 1
    I'd suggest HTMLParser/html.parser, but this solution is much better in this case.
    – Jasmijn
    Jun 13, 2011 at 9:25
  • This was a useful pointer for additional research. I actually have some broken HTML to parse, so some other answers involving lxml.html also proved useful.
    – Rob Fagen
    Jun 3, 2014 at 21:50
  • it fails if html contains unquoted attrs like "<table align=center" with lxml.etree.XMLSyntaxError: AttValue: " or ' expected Feb 22, 2017 at 18:50
74

Hands down the easiest way to parse a HTML table is to use pandas.read_html() - it accepts both URLs and HTML.

import pandas as pd
url = r'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'
tables = pd.read_html(url) # Returns list of all tables on page
sp500_table = tables[0] # Select table of interest

Only downside is that read_html() doesn't preserve hyperlinks.

8
  • 1
    Awesome! Thanks.
    – Raf
    Aug 29, 2017 at 13:45
  • 2
    Thanks! This was very quick and easy. Jan 3, 2018 at 5:54
  • Not a good way for tables containing rowspan and colspan! Aug 8, 2018 at 10:08
  • 2
    @JohnStrood Looking forward to reading your answer on how to handle rowspan and colspan 👍 Aug 8, 2018 at 23:19
  • 1
    @tommy.carstensen Ah! I used bs4 to build an element tree, and traversed through the elements to break row-spanned column-spanned cells into constituent cells. Aug 9, 2018 at 6:48
35

Sven Marnach excellent solution is directly translatable into ElementTree which is part of recent Python distributions:

from xml.etree import ElementTree as ET

s = """<table>
  <tr><th>Event</th><th>Start Date</th><th>End Date</th></tr>
  <tr><td>a</td><td>b</td><td>c</td></tr>
  <tr><td>d</td><td>e</td><td>f</td></tr>
  <tr><td>g</td><td>h</td><td>i</td></tr>
</table>
"""

table = ET.XML(s)
rows = iter(table)
headers = [col.text for col in next(rows)]
for row in rows:
    values = [col.text for col in row]
    print(dict(zip(headers, values)))

same output as Sven Marnach's answer...

3
  • 1
    +1 because it allows using cElementTree instead of ElementTree which is considerably faster than lxml if large number of tables are involved
    – Cerno
    Apr 6, 2016 at 16:00
  • 1
    I have a web page saved from wikipedia. How can I specify to ET which table to parse and fetch data ? Is it possible by table name or table id ?
    – Massimo
    May 1, 2017 at 14:31
  • 1
    also, <tbody> and <thead> don't work. see stackoverflow.com/q/49286753/8929814 Mar 14, 2018 at 21:10
21

If the HTML is not XML you can't do it with etree. But even then, you don't have to use an external library for parsing a HTML table. In python 3 you can reach your goal with HTMLParser from html.parser. I've the code of the simple derived HTMLParser class here in a github repo.

You can use that class (here named HTMLTableParser) the following way:

import urllib.request
from html_table_parser import HTMLTableParser

target = 'http://www.twitter.com'

# get website content
req = urllib.request.Request(url=target)
f = urllib.request.urlopen(req)
xhtml = f.read().decode('utf-8')

# instantiate the parser and feed it
p = HTMLTableParser()
p.feed(xhtml)
print(p.tables)

The output of this is a list of 2D-lists representing tables. It looks maybe like this:

[[['   ', ' Anmelden ']],
 [['Land', 'Code', 'Für Kunden von'],
  ['Vereinigte Staaten', '40404', '(beliebig)'],
  ['Kanada', '21212', '(beliebig)'],
  ...
  ['3424486444', 'Vodafone'],
  ['  Zeige SMS-Kurzwahlen für andere Länder ']]]
3
  • 5
    Awesome parser !!
    – Naive
    Nov 28, 2017 at 7:57
  • 1
    neat indeed. It will break if some td have a colspan though
    – mr.bjerre
    Sep 3, 2021 at 9:03
  • 1
    @mr.bjerre PR welcome ;-)
    – schmijos
    Jan 11 at 8:16

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.