0

I am trying to click load more button infinite time until load all products. But my load more button not working. here is my code:

url = 'https://www.pursemall.ru/vuitton/damier-azur'
driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url)


try:
    for i in range(1000):
     load_more_button = driver.find_element_by_xpath("//span[text()='Load More Products']")
     load_more_button.click()
except:
    pass
    print("task load more button completed") 

why my load more button not working? where I am doing mistake?

1
  • Try to inspect the element and copy its CSS selector or XPath
    – crissal
    Sep 24, 2021 at 21:50

2 Answers 2

1

What you can do is the below. Each call inside the loop will fetch a set of products. You can scrape each response and collect the data.

In the browser do F12 -> Network -> XHR and see how the browser does the HTTP call every time you hit Load More Products

import requests


for page in range(1,4):
    url = f'https://www.pursemall.ru/vuitton/damier-azur?page={page}'
    print(url)
    r = requests.get(url)
    print(r.status_code)
0
0

I'm not 100% sure why your original code wasn't functioning, but I was able to get it working by copying the full XPATH of the button instead of what you have in your code above.

I also used WebDriverWait and sleep() to wait a few seconds until the button was loaded after each click, since that's good practice for web scrapers. Perhaps the issue was a combination of the two :)

url = 'https://www.pursemall.ru/vuitton/damier-azur'
driver = webdriver.Chrome(PATH) # PATH is where you have chromedriver stored on your machine
driver.maximize_window()
driver.get(url)

try:
    for i in range(1000):
        load_more_button = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
            (By.XPATH, '/html/body/div[1]/div/div[3]/div[3]/span/a/span')))
        sleep(3)
        load_more_button.click()
except:
    pass
    print("task load more button completed")

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.