2

I'm trying to scrape data from NYSE's website, from this URL:

nyse = http://www1.nyse.com/about/listed/IPO_Index.html

Using requests, my I've set my request up like this:

page = requests.get(nyse)
soup = BeautifulSoup(page.text)

tables = soup.findAll('table')
test = pandas.io.html.read_html(str(tables))

However, I keep getting this error

'ValueError: stat: path too long for Windows'

I don't understand how to interpret this error, and furthermore, solve the problem. I've seen one other posting on this area (Copy a file with a too long path to another directory in Python) but I don't fully understand the workaround, and am not sure which path is the problem in this case.

The error is getting thrown at the test = pandas.io.... line but there isn't a clear definition of path, where I'm storing the table locally. Do I need to use pywin32? Why does this error only show for some URLs and not others? How do I solve this problem?

For reference, I'm using python 3.4

Update: The error only appears with the nyse website, and not for others that I'm also scraping. In all cases, I'm doing the str(tables) conversion.

1 Answer 1

0

The pandas read_html method accepts urls, files, or raw HTML strings as its first argument. It definitely looks like it's trying to interpret the str(tables) argument as a URL -- which would of course be quite long and overrun whatever limit Windows apparently has.

Are you certain that str(tables) produces raw, parseable HTML? Tables looks like it would be represented as a list of abstract node objects -- it seems likely that calling str() on this would not produce what you're looking for.

1
  • The output shows raw HTML, although the HTML header tags aren't included. However, the script works for the other websites that I'm trying, just not for the NYSE. One characteristic is that the NYSE is the only page that has more than one table. Could this be tripping up the pandas.io?
    – ding
    Jul 30, 2014 at 13:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.