51,383
questions
0
votes
0
answers
16
views
How to solve 'StaleElementReferenceException: Message: stale element reference: stale element not found'?
Here's my code:
elements = driver.find_elements(By.CLASS_NAME, 'name')
for i in elements:
i.click()
driver.back()
Error:
selenium.common.exceptions.StaleElementReferenceException: Message: ...
0
votes
0
answers
10
views
Is DNS caching something to consider when web scraping?
I was wondering, if I scrape one site every few seconds, Is DNS caching something to set up to avoid a lot of DNS lookup or is done automatically?
Practical example: I have a script that every few ...
-1
votes
0
answers
34
views
navigating through php and downloading file with python [closed]
I am trying to download a csv file from this website : https://candhis.cerema.fr/_public_/campagne.php
As you can see the website is a .php.
Once you open the link you still need to perform a few ...
-3
votes
0
answers
21
views
Best way to filter and sort HTML content of many website pages on the fly [closed]
So I have rather long (thousands) list of URLs that I have spidered.
But I need to sort them, which is the hard bit. Let’s say for example I wanted to sort out any URL that had $100 in the body text.
...
-2
votes
0
answers
16
views
Php curl V.S cmd curl: i always get 403 response status code when i use php curl, but 200 using cmd curl [closed]
I am trying to web scrape the document of a website. In these days' test, I found that this website's machanism is really interesting.
if i change the user-agent in the cmd curl or postman from the '...
-1
votes
0
answers
33
views
JS only works after selecting an element through the developer screen [closed]
I am using document.querySelector('#combo_inptxt').click() to select a dropdown menu, however JS does not recognize the menu until I open the developer and select an element. The dropdown is visually ...
-5
votes
0
answers
32
views
The output of my code does not match these expected outputs [closed]
I need a solution to this
# **main.py**
from compare_prices import compare_prices
laughs_coconut = 'https://scrape-sm1.github.io/site1/COCONUT%20market1super.html'
glomark_coconut = 'https://glomark....
0
votes
1
answer
60
views
Selenium ChromeDriver asking to set default search engine on startup
When Selenium opens ChromeDriver. ChromeDriver asks you to select the default search engine,and won't let me access the website I need, even if I select a search engine every time it runs, it asks for ...
-1
votes
0
answers
8
views
Archive scrape api query params
When I explore the archive website, I can see the Books category, How can I use scrape api to scrapy all books from archive.
And when I use the scrape api, the response columns cannot match the ...
1
vote
0
answers
20
views
Problem capturing requests in Rosetta Stone B1 with Node.js
Description of the problem:
I found a Node.js script to interact with Rosetta Stone Fluency Builder (B1). Previously, in the Foundations program (A1-A2), I used to capture "graphic" and &...
0
votes
1
answer
27
views
selenium's driver gets wrong page in python
I am trying to scrape certain odds for a football tournament. To this end I wrote a piece of code which first generates the exact link I want and then loads the corresponding page. The problem is, the ...
2
votes
1
answer
33
views
Failed to produce a JSON response containing a phone number based on a license number from a webpage using the requests module
I've created a script to fetch a phone number based on a license number from this webpage, using Python with the requests module. The script is supposed to produce a JSON response containing the phone ...
0
votes
0
answers
9
views
Optimizing JavaScript Code for Better Performance in High-Load Applications
I am developing a high-load web application where performance is critical. I've noticed some lag in the front-end, especially with dynamic content updates.
What I Tried:
Used requestAnimationFrame for ...
0
votes
1
answer
31
views
I'm struggling to get a .csv file from a Git-hub post using python [duplicate]
I'm trying to get a .csv file from this git post: link.
So far i tried screaping with:
import requests
import pandas as pd
from bs4 import BeautifulSoup as bs4
url = 'https://github.com/hiring-lab/...
0
votes
0
answers
28
views
How to Maintain a Consistent Location on Amazon While Scraping with Proxy Rotation in Playwright?
I am creating a script to scrape Amazon products, and I am using proxy rotation. The problem I am facing is that when Playwright creates the browser and opens the URL, it takes the location of the ...
-1
votes
0
answers
11
views
How to properly upload a file with Automa?
In Automa, I am uploading a big files that uploads 5 to 30 minutes, after that web page changes. How to click elements on a new page, after it is loaded?
I tried a while loop and 'Element Exists', and ...
0
votes
1
answer
40
views
Using rvest to webscrap a website partially protected by a login
I want to scrap some data from a web page, but this page is partially protected with username/password.
And i would like to scrap data that is only available after i logged in. I found how to navigate ...
0
votes
0
answers
24
views
Tweet Scraped Data Disorganized
So, I've been recently using Twitter's backend API, which is used to display tweet/user data on a webpage to scrape and try and find a user's latest tweet. The main issue I'm having with this is that ...
0
votes
0
answers
26
views
Python script with selenium works at local run, but doesn't on google colab
I'm working on a webscrapping script what function is to quote flight tickets on 'https://www.google.com/travel/flights', it works at my local machine, but when I try to run on Google Colab, it ...
1
vote
3
answers
42
views
Extract html table with pagination. The URL doesnt change when changing pages
I want to extract the table as attached form this link: https://www.rfi.it/en/stations.html.
But I can extract the data from page 1. I need to extract the data of all pages.
Can you guys help me? ...
0
votes
0
answers
12
views
Selenium python scraping more button for profiles
im trying to scrape linkedin profiles - to press the button more but i cannot seem to suceed at it
I tried many ways, one of them being -
def click_more_actions_button(driver):
try:
# Wait ...
0
votes
0
answers
23
views
JSONDecodeError when sending GraphQL query with Scrapy - Possible formatting issue?
I'm working on a Scrapy spider that sends a GraphQL query to an endpoint. I'm encountering a JSONDecodeError when trying to parse the response, which indicates that the response body is empty, I ...
-2
votes
0
answers
33
views
How to scrape posts from Reddit? [duplicate]
I'm having trouble scraping data from Reddit and I'm hoping someone here can help!
I've never done this before, so I'm not sure I have the basics correct. However, my code does work when I set target ...
2
votes
1
answer
47
views
How to navigate between pages using Selenium?
I'm trying to webscrape this website with car listings - cargurus.com. Specifically, I'm trying to scrape details from ferraris on sale. I've been able to get it to work, at least the first page. ...
0
votes
0
answers
36
views
Scrapy with middleware never calls callback function
I have a Scrapy crawler with a callback for extracting information from doi pages (which never runs). As my sole middleware I have seleniumbase ran with uc so as to pass off as real traffic & get ...
-2
votes
0
answers
22
views
Simulate a Button Click with No URL Change in a .aspx URL using Requests? [closed]
I'm trying to scrape data from a sports website using BeautifulSoup and the requests library in Python. The website uses `.aspx` pages, and when I click a button to view different sports categories (...
1
vote
1
answer
23
views
How to get text in beautifulsoup as .innerText and not as .textContent in JS
I have an HTML file that contains text inside a p tag, something like this:
<body>
<p>Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Maecenas sed mi lacus.
...
-1
votes
0
answers
24
views
no error in data scraping but not printing the content , tried every way of scraping ,didn't show any error but not getting the content [closed]
was trying to scrape data from web site using beautiful soup and then tried every other method ,its not showing any error put is not printing the content or data
I am expecting some solution to this, ...
0
votes
1
answer
33
views
scrape a dropdown list using playwright
I'm struggling to find a way to click on the "All" option in a dropdown list and scrape all the content inside that page. I have come across a few posts but they're a little different from ...
-3
votes
0
answers
70
views
How to scrape Etsy Product Reviews
I am unable to switch the review page on Etsy product reviews. I am only able to scrape the first page. I tried using the code below to scrap the Etsy product reviews but i was only able to scrape the ...
-1
votes
0
answers
20
views
Web Scraping data from SSRN [closed]
I'm looking to scrap data from the SSRN site for priming topic
Things to extract from the scraped papers:
• Name of the paper
• Authors
• Date
• URL
• Statistics from each study (like means, ...
0
votes
1
answer
52
views
Playwright - scraping eBay deals
from playwright.sync_api import Playwright, sync_playwright
with sync_playwright() as playwright:
chromium = playwright.chromium
browser = chromium.launch()
context = browser.new_context()...
0
votes
1
answer
29
views
Node.js library that just loads DOM and scripts but can handle async
I am trying to load a page to scrape that loads the data I need only after multiple recursive async script loads. But due to the count of pages and items, can't use things like Puppeteer or Selenium.
...
0
votes
1
answer
76
views
Need an explanation for a web scraping lambda function in Python
I'm doing Web Scraping in Python and I've found this :
products = soup.find_all('li')
products_list = []
for product in products:
name = product.h2.string
price = product.find('p', string=...
0
votes
0
answers
29
views
xpath to YouTube Subscribers is not working for me
I want to get the number of subscribers of a YouTube Channel into a Google Sheet without using the YouTube API.
I want to export youtube subscribers count in Google Sheets.
In a Google Sheet I tried ...
1
vote
2
answers
60
views
Getting correct selector using Puppeteer
here's the HTML code.
<div class="list-row">
<div class="list-item">
<div class="imgframe">
<div class="img-wrap"&...
1
vote
1
answer
40
views
Web scraping with Python BeautifulSoup
I would like to collect data by web scraping with Python BeautifulSoup from the website for my data analysis project.
the data I want to collect from the website;
date: 06.07.2027
Stage: Berghain, ...
0
votes
1
answer
67
views
Scraping data from dropdown menu with no URL
I'm attempting to automate the process of downloading data for each US county from The Climate Explorer in R. The data I'm interested require the user to select from a dropdown menu, but I can't ...
1
vote
1
answer
31
views
How do I use C# HtmlAgilityPack to get out the following text from a website?
I need to get out the text "223M" from the structure below using HtmlAgilityPack in C#, how do I do it?
<div data-widget="quote-options-flow-summary" data-mode="desktop&...
-1
votes
0
answers
43
views
Can I extract a database from a php website? [closed]
There is a website https://strongmanarchives.com/athletes.php that (if you click on the link) you will see is basically a database on a webpage, I wish to download the data so that I can run code on ...
0
votes
1
answer
51
views
Error when requesting via aio http AttributeError: the 'NoneType' object does not have the 'connect' attribute
I am writing an asynchronous parser http://freelance.habr.com where I iterate through all the tasks for a specific query. I use aio http for requests. During the execution, only some tasks give an ...
0
votes
0
answers
22
views
Undetected driver breaks down after third open url
I am doing some web scraping and I had to start using undetected chromedriver. The problem is that my code used to work with the normal chromedriver, but when using the undetected one it doesn't even ...
0
votes
1
answer
57
views
One specific span tag is empty with no attributes when using Selenium/SeleniumBase
I'm attempting to scrape data of baseball prospects on MLB.com with Python. When I open a browser manually and view the source HTML, I see the JSON data with everything I need in the single attribute ...
0
votes
0
answers
25
views
LinkedIn and Twitter Not Scraping My Next.js Page
I'm facing a bit of a challenge with getting LinkedIn and Twitter to properly scrape and preview one of my pages built with Next.js. Here’s what’s happening:
The Issue:
LinkedIn Post Inspector: When ...
0
votes
0
answers
29
views
scrape link for jwplayer calculated with JS using python
I'm trying to scrape video link (m3u8) from this website:
https://deaddrive.xyz/embed/fa31e
While inspecting the page, I realized that the link is calculated on the fly using JS in the function:
&...
2
votes
2
answers
237
views
I can't get Selenium Chrome to work in Docker with Python
I have a classic "it works on my machine" problem, a web scraper I ran successfully on my laptop, but with a persistent error whenever I tried and run it in a container.
My minimal ...
1
vote
3
answers
70
views
403 error when trying to web scrape online job sites using python
I am trying to web scrape online job offering web sites for my coursera project.
I keep getting a 403 error which, after I searched for its meaning online, I found out that it means that the web site ...
0
votes
0
answers
57
views
Recaptcha is mandatory error when try to bypass a recaptcha
I am trying to login the following site: https://agendamentos.mne.gov.pt/en/login using 2captcha and I implemented the codes correctly. The captcha is solved and sent to me, but when I click on the ...
0
votes
0
answers
12
views
How to resolve Goose3 connection error 10054?
I run a script on daily basis using a cronjob and here we are using g.extract to extract articles and my script fails 2 out of 5 times with an error : Error occurred: ('Connection aborted.', ...
0
votes
1
answer
25
views
Web scraping using Apify with load more button
I am trying to scrape news articles from the website here using Apify-Smart Article Extractor. However, the webpage contains a 'load more' button which needs to be clicked in order to reveal required ...