-1

I finally got it to create the csv file but for some reason it creates the headers but never fills in the data

import requests
from bs4 import BeautifulSoup
import csv

url = "http://www.scsotx.org/jail-booking"
r = requests.get(url)

soup = BeautifulSoup(r.content, 'html5lib')

inmate_data =[]

table = soup.find('div', attrs = {'class':'sex-offender-info'})

for row in table.findAll('div', attrs = {'class':'jail-content'}):
    jaildata = {}
    jaildata['Name'] = row.h4.text
    jaildata['Agency'] = row.p.text
    inmate_data.append(jail-content)


with open('C:\\Users\Cale\Desktop\jail\inmate_data.csv', 'w') as f:
    w = csv.DictWriter(f,['Name','Agency'])
    w.writeheader()
    for jaildata in inmate_data:
        w.writerow(jaildata)

its supposed to parse the html data and then append the csv file

0

3 Answers 3

2

Here's your code fixed:

for row in table.findAll("figcaption", attrs={"class": "jail-content"}):
    jaildata = {}
    jaildata["Name"] = row.h4.text
    jaildata["Agency"] = row.p.text
    inmate_data.append(jaildata)

The data you were looking for was inside <figcaption> instead of <div>, and also there was a typo with jail-content instead of jaildata when trying to append.

4
  • Please add an explanation on where the mistake was.
    – Irfanuddin
    Commented Jul 3, 2019 at 4:31
  • I've added a brief explanation of the two mistakes.
    – jacalvo
    Commented Jul 3, 2019 at 4:33
  • thank you the typo was so stupid to miss but the figcaption makes sense, any idea why it still only returns the one result? should i use findAll in the table? Commented Jul 3, 2019 at 4:50
  • changed table = soup.find("body") Commented Jul 3, 2019 at 5:00
0

Although I strongly recommend you to use selenium for this sort of thing, this is what you can do to improve the workings of your scraping:

import requests
import pandas as pd
from bs4 import BeautifulSoup


class ScrapeJail:
    def __init__(self, url: str = "http://www.scsotx.org/jail-booking"):
        self.url = url

    def get_table(self):
        response = requests.get(self.url)
        soup = BeautifulSoup(response.content, 'html5lib')
        raw_data = soup.find('div', attrs={'class': 'sex-off-box'})
        data_contents = raw_data.findAll('div', attrs={'class': "medium-4 small-6 columns"})
        data = []
        for i, _ in enumerate(data_contents):
            person_data_tags = data_contents[i].findAll('div', attrs={'class': "sex-offender-info"})
            person_data_jail = person_data_tags[0].findAll(attrs={'class': "jail-content"})
            person_data = person_data_jail[0].findChildren()
            person_dict = {}
            for tag in person_data:
                person_text = tag.text
                try:
                    points = person_text.index(':')
                    person_dict[person_text[:points]] = person_text[points + 1:]
                    data.append(person_dict)
                except ValueError:
                    pass
        return data

    def data_frame(self):
        return pd.DataFrame(self.get_table())

    def export_csv(self, file_name:str):
        df = self.data_frame()
        df.to_csv(file_name)

Not perfect, the charges are not perfect, but simply do a

data = ScrapeJail()

csv = data.export_csv('file_name.csv') 
-1
for row in table.findAll('div', attrs = {'class':'jail-content'}):
jaildata = {}
jaildata['Name'] = row.h4.text
jaildata['Agency'] = row.p.text
inmate_data.append(jail-content)

If you look at this block, the variable jail-content on the last line isn't being declared at all. I'm assuming you want to use jaildata?

1
  • thanks for catching the typo guys. the figcaption change fixed that part but something is still wrong its not going through the whole page only returns the one result Commented Jul 3, 2019 at 4:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.