1

I'm trying to scrape that website for the Captcha image link.

Using browser inspect element it's already appear but upon scraping it's not shown.

My target were to getting the img

Below is my code which i tried with it.

import requests
from bs4 import BeautifulSoup

with requests.Session() as s:
    url = "https://myurl.com/"
    r = s.get(url)
    soup = BeautifulSoup(r.content, "html.parser")
    for item in soup.findAll("img"):
        print(item)
4
  • The image is loaded with xhr. You can see in the page source the javascript code that loads it. Nov 19, 2019 at 10:34
  • It's an event. Normal soup.findAll won't find possibly. Nov 19, 2019 at 10:35
  • are there a way to extract it ?
    – Anna Plym
    Nov 19, 2019 at 10:35
  • use selenium webdriver with python Nov 19, 2019 at 10:36

2 Answers 2

1

If you go to 'NetWork' tab you will get below link which returns the captcha image in JSON format. You don't need Selenium for that.

https://example.com/site/captcha/refresh/1/?_=1574163338269

You need to convert response into JSON and then get the url key val.

import requests

with requests.Session() as s:
    url = "https://example.com/site/captcha/refresh/1/?_=1574163338269"
    r = s.get(url, verify=False)
    img = r.json()
    print(img['url'])

NetworkTab

screenshot[1]

1
  • the point that this is 1/?_=1574163338269 changeable on each request/refresh
    – Anna Plym
    Nov 19, 2019 at 21:30
0

Like others have said, selenium will help load the img allowing you to scrape it.

from selenium import webdriver
from bs4 import BeautifulSoup
import time

browser = webdriver.Firefox()
url = 'https://myurl.com/'
browser.get(url)
time.sleep(10) # wait 10 seconds for the captcha to load
html = browser.page_source
soup = BeautifulSoup(html,features='html.parser')

imgs = soup.find_all('img')
for img in imgs:
    print(img)

Returns:

<img alt="" id="yw1" src="/site/captcha/v/5dd3ccb47dd88/"/>

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.