I am currently looping through URL's and grabbing data while visiting/crawling websites.
Sometimes a website will have an unreasonably long loading time where no errors are made, but will not fully load to allow chromedriver/urlopen to complete/continue with the script and just stays in limbo.
Dynamically testing for presence of element does not work in this case as the page wont completely load, and pages are not all the same to test for fixed elements (not even abundant tags like html or h1 tags etc).
Basically I am looking for a code that will continue to the next iteration the loop after "x" seconds if the page dosent load
Currently using Selenium (chromedriver) and Beautifulsoup (BS4).
def get_emails_from_list(links):
email=[]
for link in links:
driver.get(link)
html=driver.page_source
try:
raw = BeautifulSoup(html, 'html.parser').get_text()
emails = re.findall(r'[\w\.-]+@[\w\.-]+', raw)
for em in emails:
if em not in email:
email.append(emails)
except:
emails = re.findall(r'[\w\.-]+@[\w\.-]+', str(html))
for em in emails:
if em not in email:
email.append(emails)
try:
email2=list(itertools.chain(*email))
except:
email2=email
return email2