0

I am new to beautiful soup and am trying to figure out how to pull a website from a nested array. The website can be found twice under the "track-visit-website" class.

This is NOT a duplicate of the question asking about how to pull hrefs. I've done that successfully on this page. I am trying to isolate the actual company website.

I've tried several codes, but can't get it to work. Here is an example:

print(item.contents[2].find_all("a", {"class": "track-visit-website"})[0].a)

The site is YP.com Septic Search

Here's the code from the one of the items on the site:

<div class="info">
<h3 class="n">
<div class="info-section info-primary">
<p class="adr" itemprop="address" itemtype="http://schema.org/PostalAddress" itemscope="">
<span class="street-address" itemprop="streetAddress">2806 Farview Dr</span>
<span class="locality" itemprop="addressLocality">Fort Collins, </span>
<span itemprop="addressRegion">CO</span>
<span itemprop="postalCode">80524</span>
</p>
<div class="phones phone primary" itemprop="telephone">(970) 829-0852</div>
</div>
<div class="info-section info-secondary">
<div class="categories">
<div class="links">
<a class="track-visit-website" data-analytics="{"click_id":6,"act":2,"dku":"http://www.affordablesepticanddraincleaning.com","FL":"url","TL":"off","target":"website","LOC":"http://www.affordablesepticanddraincleaning.com"}" target="_blank" rel="nofollow" href="http://www.affordablesepticanddraincleaning.com" data-impressed="1">Website</a>
<a class="track-map-it directions" data-analytics="{"click_id":13,"target":"website","act":4}" href="/listings/1000775636908/directions" data-impressed="1">Directions</a>
<a class="track-more-info" data-analytics="{"click_id":7,"target":"moreInfo","act":1,"FL":"list"}" href="/fort-collins-co/mip/affordable-septic-drain-cleaning-llc-505109997?lid=1000775636908" data-impressed="1">More Info</a>
</div>
1
  • This isn't a duplicate of the questions referred. If I just look for 'a' or 'href', it kicks out several falses. The question has to do with dealing with nested arrays.
    – pekasus
    May 20, 2015 at 12:49

1 Answer 1

0

Copy this code snippet to a python file and run it

import re

content = """
<div class="info">
<h3 class="n">
<div class="info-section info-primary">
<p class="adr" itemprop="address" itemtype="http://schema.org/PostalAddress" itemscope="">
<span class="street-address" itemprop="streetAddress">2806 Farview Dr</span>
<span class="locality" itemprop="addressLocality">Fort Collins, </span>
<span itemprop="addressRegion">CO</span>
<span itemprop="postalCode">80524</span>
</p>
<div class="phones phone primary" itemprop="telephone">(970) 829-0852</div>
</div>
<div class="info-section info-secondary">
<div class="categories">
<div class="links">
<a class="track-visit-website" data-analytics="{"click_id":6,"act":2,"dku":"http://www.affordablesepticanddraincleaning.com","FL":"url","TL":"off","target":"website","LOC":"http://www.affordablesepticanddraincleaning.com"}" target="_blank" rel="nofollow" href="http://www.affordablesepticanddraincleaning.com" data-impressed="1">Website</a>
<a class="track-map-it directions" data-analytics="{"click_id":13,"target":"website","act":4}" href="/listings/1000775636908/directions" data-impressed="1">Directions</a>
<a class="track-more-info" data-analytics="{"click_id":7,"target":"moreInfo","act":1,"FL":"list"}" href="/fort-collins-co/mip/affordable-septic-drain-cleaning-llc-505109997?lid=1000775636908" data-impressed="1">More Info</a>
</div>
"""

websites = set(re.findall(r'http://[a-zA-Z0-9\.]*\.[a-z]{2,}',content)) # find all urls in the content
websites = list(websites)

print(websites)  # or in python2 => print websites

Now find a way to incorporate that into your code, get the html, save it as content, regex it and save to file

Web scraping you have to know regex

read up on regex, a good tutorial is here regex tutorial

2
  • Thanks for the link. I read through it, but it will take a few more times to sink in. I tried that code and it doesn't return anything. I tried content = item.contents[1].find_all("a", {"class": "track-visit-website"})[0] and got it to return the section. Is there no way to select the "LOC" attribute directly?
    – pekasus
    May 20, 2015 at 21:27
  • the regex code i sent has no association with beautiful soup, just take the the html code from the one of the items on the site and save it as html, i've updated my answer May 21, 2015 at 5:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.