Sorry for the confusing title. I'm a complete beginner in Python and don't even know the language for asking this question.
I'm trying to do some data scrubbing of the website Box Office Mojo. I'm looking to create a csv file that pulls the table for each countries top box office hits for each year (see https://www.boxofficemojo.com/intl/austria/yearly/?yr=2019&p=.htm
I've been able to use BeautifulSoup to grab a particular page and parse it:
pip install bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
##Creating a variable for the URL I want to use
my_url = 'https://www.boxofficemojo.com/intl/austria/yearly/?yr=2019&p=.htm'
uReq(my_url)
### Opening up connecting, grabbing the page, closing the connection
uClient = uReq(my_url)
page_html = uClient.read() ###Don't run page_html because it may crash the client
uClient.close()
##Using BeuatifulSoup to read the html page and parsing it
page_soup = soup(page_html,"html.parser")
I've been able to isolate the particular line of html that I want to grab the information from:
container_odd = page_soup.findAll("tr", {"bgcolor":"#ffffff"})
container_even = page_soup.findAll("tr", {"bgcolor":"#f4f4ff"})
I can see that it's grabbed the text I want and can get it without all the html formatting.
>>>>container_even[0]
<tr bgcolor="#f4f4ff"><td align="center"><font size="2">2</font></td>
<td><font size="2">How to Train Your Dragon: The Hidden World</font></td>
<td><font size="2">UPI</font></td>
<td align="right"><font size="2"><b>$2,701,010</b></font></td>
<td align="center"><font size="2">2/8</font></td>
</tr>
>>> container_odd[0].text
'1\nCaptain Marvel\nDisney\n$3,221,398\n3/7\n'
I can also call the first 'td item'(I don't know what to call this) in the row:
>>> container_odd[0].td.text
'1'
I cannot for the life of me figure out how to pull that second 'td item' in the row, i.e., 'Captain Marvel'.
What I want to do now is create a loop that will pull each individual td item from each row in container_even/odd to put in a list.
So, I want a rank_list to contain '1, 2, 3, 4...' (Or rather '1,3,5...' and '2,4,6...' because of the code in this case)
and a title list to contain 'Captain Marvel, How to train your dragon....'
What I can't figure out is how to pull that second 'td item' and then how to loop across that item in each row?
I've tried to pull at least the rank item list:
rank = td[]
print("rank: " + rank)
But just got a syntax error
File "scraper.py", line 25
rank = td[]
^
SyntaxError: invalid syntax
Any help getting the loop created or at least figuring out how to pull that second td item would be much appreciated!