0

I'm writing a little scraper. Here's the code so far.

from urllib import urlopen
from BeautifulSoup import BeautifulSoup
import re

soup = BeautifulSoup(
    urlopen('http://www.high-rely.com/HR3/includes/ProductFamily.php').read()
    )

links = soup.findAll('a', 'visible_link')

hrefs = ['www.high-rely.com' + relative for relative in [x['href'] for x in links]]

subpages = map(BeautifulSoup, [urlopen(x).read() for x in hrefs])

When I run it though, I get the following error.

Traceback (most recent call last):
  File "C:/Users/josh.SCL/Desktop/Scraper.py", line 13, in <module>
    subpages = map(BeautifulSoup, [urlopen(x).read() for x in hrefs])
  File "C:\Python27\lib\urllib.py", line 84, in urlopen
    return opener.open(url)
  File "C:\Python27\lib\urllib.py", line 205, in open
    return getattr(self, name)(url)
  File "C:\Python27\lib\urllib.py", line 461, in open_file
    return self.open_local_file(url)
  File "C:\Python27\lib\urllib.py", line 475, in open_local_file
    raise IOError(e.errno, e.strerror, e.filename)
IOError: [Errno 2] The system cannot find the path specified: 'www.high-rely.com\\HR3\\includes\\products\\5MinOverview.php'

If I loop through hrefs, I get this.

www.high-rely.com/HR3/includes/products/5MinOverview.php
www.high-rely.com/HR3/includes/products/10MinOverview.php
www.high-rely.com/HR3/includes/products/30MinOverview.php
www.high-rely.com/HR3/includes/HighRely/HighRely.php
www.high-rely.com/HR3/includes/HighRely/HighRely.php
www.high-rely.com/HR3/includes/RAIDFrame/RAIDFrame.php
www.high-rely.com/HR3/includes/RAIDFrame/RAIDFrame.php
www.high-rely.com/HR3/includes/MPac/MPac.php
www.high-rely.com/HR3/includes/MPac/MPac.php
www.high-rely.com/HR3/includes/BNAS/BNAS-HRS201.php
www.high-rely.com/HR3/includes/announcements.php

Which is correct. What's going on here?

1
  • try urlparse.urljoin('http://www.high-rely.com', relative). And urllib2.urlopen instead of urllib.urlopen.
    – jfs
    Nov 17, 2011 at 22:01

1 Answer 1

3

You forgot to write http://:

hrefs = ['http://www.high-rely.com' + relative for relative in [x['href'] for x in links]]
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.