I have a list of elements from a web page that I want to iterate through. I want to click on each element then grab data from the page that element linked to. I need to save this data to a data frame. The problem here is that only the data from the first element is being entered into the data frame. The data frame should have information from 84 different properties.
Why does the data frame only have the data from the first element in properties
?
List of elements:
print(properties[0:5])
[<selenium.webdriver.remote.webelement.WebElement (session="93C94F07-6E83-4FCF-BC81-C1094B2D57FC", element="node-E0326EE3-BB10-4ED5-A705-257D151516B4")>,
<selenium.webdriver.remote.webelement.WebElement (session="93C94F07-6E83-4FCF-BC81-C1094B2D57FC", element="node-07A8FADF-A8A3-4A98-94D6-3F59493685DE")>,
<selenium.webdriver.remote.webelement.WebElement (session="93C94F07-6E83-4FCF-BC81-C1094B2D57FC", element="node-1494D76E-B621-45CC-B808-F0D66B1CE9D6")>,
<selenium.webdriver.remote.webelement.WebElement (session="93C94F07-6E83-4FCF-BC81-C1094B2D57FC", element="node-1B28849F-78F9-48F1-8A35-2DD806DA647B")>,
<selenium.webdriver.remote.webelement.WebElement (session="93C94F07-6E83-4FCF-BC81-C1094B2D57FC", element="node-378D2946-54EF-4570-8070-CC3D25D1F83B")>]
For loop:
for index, prop in enumerate(properties):
try:
prop.click()
url = driver.current_url
html_doc = urllib.request.urlopen(url)
soup = BeautifulSoup(html_doc, 'html.parser')
details = soup.find_all('p')
details = [i.get_text() for i in details][5:20]
details = [i.split(':') for i in details]
columns = [i[0] for i in details]
data = [[i[1][1:] for i in details]]
df = df.append(data)
if index <1:
df = pd.DataFrame(data, columns=columns)
else:
df = df.append(data)
except exceptions.StaleElementReferenceException:
pass
Dataframe (should be 84 rows instead of 1)
print(df)
Property Address Price Rent Type Class Square Footage Bedroom Bathroom Year Built Taxes Insurance Operating Expense Down Payment Cash on Cash ROI Monthly Cash Flow
100 Example Dr $75,000 $700 SFR B 630 1 1.0 1925 $1,020 $400 N/A $15,000 18% $186
.append()
modifies its object in place and returnsNone
. What happens if you usedf.append(data)
instead ofdf = df.append(data)
? – Tim Pietzcker Oct 30 '20 at 21:54