47

I'm trying to scrape the data from the coins catalog.

There is one of the pages. I need to scrape this data into Dataframe

So far I have this code:

import bs4 as bs
import urllib.request
import pandas as pd

source = urllib.request.urlopen('http://www.gcoins.net/en/catalog/view/45518').read()
soup = bs.BeautifulSoup(source,'lxml')

table = soup.find('table', attrs={'class':'subs noBorders evenRows'})
table_rows = table.find_all('tr')

for tr in table_rows:
    td = tr.find_all('td')
    row = [tr.text for tr in td]
    print(row)                    # I need to save this data instead of printing it 

It produces following output:

[]
['', '', '1882', '', '108,000', 'UNC', '—']
[' ', '', '1883', '', '786,000', 'UNC', '~ $3.99']
[' ', " \n\n\n\n\t\t\t\t\t\t\t$('subGraph55337').on('click', function(event) {\n\t\t\t\t\t\t\t\tLightview.show({\n\t\t\t\t\t\t\t\t\thref : '/en/catalog/ajax/subgraph?id=55337',\n\t\t\t\t\t\t\t\t\trel : 'ajax',\n\t\t\t\t\t\t\t\t\toptions : {\n\t\t\t\t\t\t\t\t\t\tautosize : true,\n\t\t\t\t\t\t\t\t\t\ttopclose : true,\n\t\t\t\t\t\t\t\t\t\tajax : {\n\t\t\t\t\t\t\t\t\t\t\tevalScripts : true\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t} \n\t\t\t\t\t\t\t\t});\n\t\t\t\t\t\t\t\tevent.stop();\n\t\t\t\t\t\t\t\treturn false;\n\t\t\t\t\t\t\t});\n\t\t\t\t\t\t", '1884', '', '4,604,000', 'UNC', '~ $2.08–$4.47']
[' ', '', '1885', '', '1,314,000', 'UNC', '~ $3.20']
['', '', '1886', '', '444,000', 'UNC', '—']
[' ', '', '1888', '', '413,000', 'UNC', '~ $2.88']
[' ', '', '1889', '', '568,000', 'UNC', '~ $2.56']
[' ', " \n\n\n\n\t\t\t\t\t\t\t$('subGraph55342').on('click', function(event) {\n\t\t\t\t\t\t\t\tLightview.show({\n\t\t\t\t\t\t\t\t\thref : '/en/catalog/ajax/subgraph?id=55342',\n\t\t\t\t\t\t\t\t\trel : 'ajax',\n\t\t\t\t\t\t\t\t\toptions : {\n\t\t\t\t\t\t\t\t\t\tautosize : true,\n\t\t\t\t\t\t\t\t\t\ttopclose : true,\n\t\t\t\t\t\t\t\t\t\tajax : {\n\t\t\t\t\t\t\t\t\t\t\tevalScripts : true\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t} \n\t\t\t\t\t\t\t\t});\n\t\t\t\t\t\t\t\tevent.stop();\n\t\t\t\t\t\t\t\treturn false;\n\t\t\t\t\t\t\t});\n\t\t\t\t\t\t", '1890', '', '2,137,000', 'UNC', '~ $1.28–$4.79']
['', '', '1891', '', '605,000', 'UNC', '—']
[' ', '', '1892', '', '205,000', 'UNC', '~ $4.47']
[' ', '', '1893', '', '754,000', 'UNC', '~ $4.79']
[' ', '', '1894', '', '532,000', 'UNC', '~ $3.20']
[' ', '', '1895', '', '423,000', 'UNC', '~ $2.40']
['', '', '1896', '', '174,000', 'UNC', '—']

But when I'm trying to save it to Dataframe and export to excel it contains just the last value:

         0
0         
1         
2     1896
3         
4  174,000
5      UNC
6        —
3
  • How did you do to save it to Dataframe ?
    – phi
    Commented May 31, 2018 at 22:04
  • Hi, phi. Good catch. I didn't mention it. I just added 2 lines more: df = pd.DataFrame(row) and df.to_excel('coins.xlsx'). The data in the for loop wa overwriting.
    – Alex
    Commented Jun 1, 2018 at 20:13
  • You can also use df['col'].str.strip('\n') to delete \n
    – abdoulsn
    Commented Oct 11, 2019 at 8:30

6 Answers 6

111

Pandas already has a built-in method to convert the table on the web to a dataframe:

table = soup.find_all('table')
df = pd.read_html(str(table))[0]
2
  • 2
    This should be the accepted answer. No point in using BeautifulSoup for this use case.
    – Sid
    Commented Jul 20, 2020 at 8:28
  • 1
    This seems right but has inconsistent behaviour loosing vast amounts of table rows.
    – avibrazil
    Commented Aug 16, 2020 at 11:59
34

Try this

l = []
for tr in table_rows:
    td = tr.find_all('td')
    row = [tr.text for tr in td]
    l.append(row)
pd.DataFrame(l, columns=["A", "B", ...])
0
18

Try:

import pandas as pd
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
table = soup.find('table', attrs={'class':'subs noBorders evenRows'})
table_rows = table.find_all('tr')

res = []
for tr in table_rows:
    td = tr.find_all('td')
    row = [tr.text.strip() for tr in td if tr.text.strip()]
    if row:
        res.append(row)


df = pd.DataFrame(res, columns=["Year", "Mintage", "Quality", "Price"])
print(df)

Output:

   Year  Mintage Quality    Price
0  1882  108,000     UNC        —
1  1883  786,000     UNC  ~ $4.03
1
  • Hi, Rakesh. Thanks for your answer. It works for me as well. I just chose the phi's answer because it boosted my work yesterday :) Cheers!
    – Alex
    Commented Jun 1, 2018 at 20:20
2

Just a head's up... This part of Rakesh's code means that only HTML rows containing text will be included in the dataframe, as the rows don't get appended if row is an empty list:

if row:
    res.append(row)

Problematic in my use case, where I wanted to compare row indexing for the HTML and dataframe tables later on. I just needed to change it to:

res.append(row)

Also, if a cell in the row is empty, it doesn't get included. This then messes up the columns. So I changed

row = [tr.text.strip() for tr in td if tr.text.strip()]

to

row = [d.text.strip() for d in td]

But, otherwise, it's working for me. Thanks :)

2

No need for Beautifulsoup at all. If you just want to extract html tables into DataFrames just use

dfs = pd.read_html(url)

With url being the actual website url (i.e. 'http://www.gcoins.net/en/catalog/view/45518').

The pandas function will parse the page automatically and return a list of dataframe objects created from the tables in the HTML code.

2

Since Pandas has a built-in parser that has a method to convert the table on the web to a dataframe, you can also use the following prettify() method on a beautifulsoup table element as an input to the pandas read_html method to get the dataframe/dataframes from the element:

table_elem = soup.find('table')
df = pd.read_html(table_elem.prettify())[0]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.