1

I faced one issue, before when I scrape multiple pages by Selenium, just use click next page button and use NoSuchElementException to stop it.

But the url I facing now is the element always exists, in the last page, if I click next page button, it just reload the current page.

Anyone Can help to solve how to stop click the same button?

items=driver.find_elements_by_class_name('item')
while True:
        try:
            #click next page
            driver.find_element_by_link_text('下一页').click()
            sleep(5)
            #scrpae data here
            items=driver.find_elements_by_class_name('item')

            for i in range(0, len(items)):

               results.append(items[i])
               print(items[i])
             
              
              
        except NoSuchElementException:
            
            break        

For the pages details you can check the picture below

Fullsize image

1
  • resource page link?
    – jia Jimmy
    Jan 14, 2019 at 9:43

2 Answers 2

2

[Edited]

You can solve it by matching current page url and next page url in next page link. if current page url matches the url in next page link then it is the last page. otherwise continue scraping.

You should have a variable where you store current page URL and when you click on next page link by selenium, you get the page url and match with previous.

This is what i am saying:

url = "https://humkinar.com.pk/"
driver.get(url)
items=driver.find_elements_by_class_name('item')
current_page_url = ""
prev_page_url = url
while True:
    try:
        driver.find_element_by_link_text('下一页').click()
        current_page_url = driver.current_url
        if current_page_url != prev_page_url:
            time.sleep(5)
            items=driver.find_elements_by_class_name('item')
            for i in range(0, len(items)):
               results.append(items[i])
               print(items[i])
            prev_page_url = current_page_url
        else:
            break
    except NoSuchElementException:
        break
3
  • would you mind show me any code for our reference? thanks in advance!
    – Yan Zhang
    Jan 14, 2019 at 8:30
  • You can get current url from selenium web driver like this print driver.current_url Jan 15, 2019 at 9:43
  • if you want complete code i can send you by editing your code. Jan 15, 2019 at 9:44
1

As i see in picture (i suppose picture you shared is of last page),check for className == 'disable' in <a class='disable'> <some text in chinese></a> and break;

UPDATE:

items=driver.find_elements_by_class_name('item')
while True:
    try:
        #click next page
        next = driver.find_element_by_link_text('下一页')
        next.click()
        sleep(5)
        #scrpae data here
        items=driver.find_elements_by_class_name('item')

        for i in range(0, len(items)):

           results.append(items[i])
           print(items[i])

           break;        
       if(next.getAttribute('class') == 'disable'){
          throw new Exception()
       }
2
  • Hi, Mr. Yogendra Porwal, Thank you very much for your answer, would you mind tell me why I get error as below: File "<ipython-input-10-ab3da9243700>", line 18 if(next.getAttribute('class') == 'disable'){ ^ SyntaxError: invalid syntax ====>you can try run this url link
    – Yan Zhang
    Jan 22, 2019 at 3:53
  • this a java syntax please use an if syntax valid for python :) Jan 25, 2019 at 7:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.