Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

How to solve 'StaleElementReferenceException: Message: stale element reference: stale element not found'?

Here's my code: elements = driver.find_elements(By.CLASS_NAME, 'name') for i in elements: i.click() driver.back() Error: selenium.common.exceptions.StaleElementReferenceException: Message: ...
Vignesh Elumalai's user avatar
0 votes
0 answers
10 views

Is DNS caching something to consider when web scraping?

I was wondering, if I scrape one site every few seconds, Is DNS caching something to set up to avoid a lot of DNS lookup or is done automatically? Practical example: I have a script that every few ...
Jhon's user avatar
  • 13
-1 votes
0 answers
34 views

navigating through php and downloading file with python [closed]

I am trying to download a csv file from this website : https://candhis.cerema.fr/_public_/campagne.php As you can see the website is a .php. Once you open the link you still need to perform a few ...
bhern's user avatar
  • 1
-3 votes
0 answers
21 views

Best way to filter and sort HTML content of many website pages on the fly [closed]

So I have rather long (thousands) list of URLs that I have spidered. But I need to sort them, which is the hard bit. Let’s say for example I wanted to sort out any URL that had $100 in the body text. ...
security_paranoid's user avatar
-2 votes
0 answers
16 views

Php curl V.S cmd curl: i always get 403 response status code when i use php curl, but 200 using cmd curl [closed]

I am trying to web scrape the document of a website. In these days' test, I found that this website's machanism is really interesting. if i change the user-agent in the cmd curl or postman from the '...
huimin chen's user avatar
-1 votes
0 answers
33 views

JS only works after selecting an element through the developer screen [closed]

I am using document.querySelector('#combo_inptxt').click() to select a dropdown menu, however JS does not recognize the menu until I open the developer and select an element. The dropdown is visually ...
Sharks  Charlatans's user avatar
-5 votes
0 answers
32 views

The output of my code does not match these expected outputs [closed]

I need a solution to this # **main.py** from compare_prices import compare_prices laughs_coconut = 'https://scrape-sm1.github.io/site1/COCONUT%20market1super.html' glomark_coconut = 'https://glomark....
B.M.M. Yasas S.'s user avatar
0 votes
1 answer
60 views

Selenium ChromeDriver asking to set default search engine on startup

When Selenium opens ChromeDriver. ChromeDriver asks you to select the default search engine,and won't let me access the website I need, even if I select a search engine every time it runs, it asks for ...
Leonardo Cremona's user avatar
-1 votes
0 answers
8 views

Archive scrape api query params

When I explore the archive website, I can see the Books category, How can I use scrape api to scrapy all books from archive. And when I use the scrape api, the response columns cannot match the ...
仲启超's user avatar
1 vote
0 answers
20 views

Problem capturing requests in Rosetta Stone B1 with Node.js

Description of the problem: I found a Node.js script to interact with Rosetta Stone Fluency Builder (B1). Previously, in the Foundations program (A1-A2), I used to capture "graphic" and &...
Af Fe's user avatar
  • 11
0 votes
1 answer
27 views

selenium's driver gets wrong page in python

I am trying to scrape certain odds for a football tournament. To this end I wrote a piece of code which first generates the exact link I want and then loads the corresponding page. The problem is, the ...
Sinusx's user avatar
  • 111
2 votes
1 answer
33 views

Failed to produce a JSON response containing a phone number based on a license number from a webpage using the requests module

I've created a script to fetch a phone number based on a license number from this webpage, using Python with the requests module. The script is supposed to produce a JSON response containing the phone ...
robots.txt's user avatar
0 votes
0 answers
9 views

Optimizing JavaScript Code for Better Performance in High-Load Applications

I am developing a high-load web application where performance is critical. I've noticed some lag in the front-end, especially with dynamic content updates. What I Tried: Used requestAnimationFrame for ...
Hamza Khalid's user avatar
0 votes
1 answer
31 views

I'm struggling to get a .csv file from a Git-hub post using python [duplicate]

I'm trying to get a .csv file from this git post: link. So far i tried screaping with: import requests import pandas as pd from bs4 import BeautifulSoup as bs4 url = 'https://github.com/hiring-lab/...
Ramiro's user avatar
  • 39
0 votes
0 answers
28 views

How to Maintain a Consistent Location on Amazon While Scraping with Proxy Rotation in Playwright?

I am creating a script to scrape Amazon products, and I am using proxy rotation. The problem I am facing is that when Playwright creates the browser and opens the URL, it takes the location of the ...
Javi Vaz's user avatar
-1 votes
0 answers
11 views

How to properly upload a file with Automa?

In Automa, I am uploading a big files that uploads 5 to 30 minutes, after that web page changes. How to click elements on a new page, after it is loaded? I tried a while loop and 'Element Exists', and ...
Mrslrts's user avatar
0 votes
1 answer
40 views

Using rvest to webscrap a website partially protected by a login

I want to scrap some data from a web page, but this page is partially protected with username/password. And i would like to scrap data that is only available after i logged in. I found how to navigate ...
BPeif's user avatar
  • 350
0 votes
0 answers
24 views

Tweet Scraped Data Disorganized

So, I've been recently using Twitter's backend API, which is used to display tweet/user data on a webpage to scrape and try and find a user's latest tweet. The main issue I'm having with this is that ...
LordTakahiro's user avatar
0 votes
0 answers
26 views

Python script with selenium works at local run, but doesn't on google colab

I'm working on a webscrapping script what function is to quote flight tickets on 'https://www.google.com/travel/flights', it works at my local machine, but when I try to run on Google Colab, it ...
Wagner simões's user avatar
1 vote
3 answers
42 views

Extract html table with pagination. The URL doesnt change when changing pages

I want to extract the table as attached form this link: https://www.rfi.it/en/stations.html. But I can extract the data from page 1. I need to extract the data of all pages. Can you guys help me? ...
Tan Phan's user avatar
0 votes
0 answers
12 views

Selenium python scraping more button for profiles

im trying to scrape linkedin profiles - to press the button more but i cannot seem to suceed at it I tried many ways, one of them being - def click_more_actions_button(driver): try: # Wait ...
Liga Lomakina ORQBAS's user avatar
0 votes
0 answers
23 views

JSONDecodeError when sending GraphQL query with Scrapy - Possible formatting issue?

I'm working on a Scrapy spider that sends a GraphQL query to an endpoint. I'm encountering a JSONDecodeError when trying to parse the response, which indicates that the response body is empty, I ...
NoOne's user avatar
  • 11
-2 votes
0 answers
33 views

How to scrape posts from Reddit? [duplicate]

I'm having trouble scraping data from Reddit and I'm hoping someone here can help! I've never done this before, so I'm not sure I have the basics correct. However, my code does work when I set target ...
baixiwei's user avatar
  • 1,041
2 votes
1 answer
47 views

How to navigate between pages using Selenium?

I'm trying to webscrape this website with car listings - cargurus.com. Specifically, I'm trying to scrape details from ferraris on sale. I've been able to get it to work, at least the first page. ...
ssennna_812's user avatar
0 votes
0 answers
36 views

Scrapy with middleware never calls callback function

I have a Scrapy crawler with a callback for extracting information from doi pages (which never runs). As my sole middleware I have seleniumbase ran with uc so as to pass off as real traffic & get ...
Cudon's user avatar
  • 1
-2 votes
0 answers
22 views

Simulate a Button Click with No URL Change in a .aspx URL using Requests? [closed]

I'm trying to scrape data from a sports website using BeautifulSoup and the requests library in Python. The website uses `.aspx` pages, and when I click a button to view different sports categories (...
niksbiedrins's user avatar
1 vote
1 answer
23 views

How to get text in beautifulsoup as .innerText and not as .textContent in JS

I have an HTML file that contains text inside a p tag, something like this: <body> <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas sed mi lacus. ...
Avraham's user avatar
  • 63
-1 votes
0 answers
24 views

no error in data scraping but not printing the content , tried every way of scraping ,didn't show any error but not getting the content [closed]

was trying to scrape data from web site using beautiful soup and then tried every other method ,its not showing any error put is not printing the content or data I am expecting some solution to this, ...
Anushka Keshri's user avatar
0 votes
1 answer
33 views

scrape a dropdown list using playwright

I'm struggling to find a way to click on the "All" option in a dropdown list and scrape all the content inside that page. I have come across a few posts but they're a little different from ...
Anh Tu Pham's user avatar
-3 votes
0 answers
70 views

How to scrape Etsy Product Reviews

I am unable to switch the review page on Etsy product reviews. I am only able to scrape the first page. I tried using the code below to scrap the Etsy product reviews but i was only able to scrape the ...
Mo H's user avatar
  • 1
-1 votes
0 answers
20 views

Web Scraping data from SSRN [closed]

I'm looking to scrap data from the SSRN site for priming topic Things to extract from the scraped papers: • Name of the paper • Authors • Date • URL • Statistics from each study (like means, ...
sam sam's user avatar
0 votes
1 answer
52 views

Playwright - scraping eBay deals

from playwright.sync_api import Playwright, sync_playwright with sync_playwright() as playwright: chromium = playwright.chromium browser = chromium.launch() context = browser.new_context()...
Swelan Auguste's user avatar
0 votes
1 answer
29 views

Node.js library that just loads DOM and scripts but can handle async

I am trying to load a page to scrape that loads the data I need only after multiple recursive async script loads. But due to the count of pages and items, can't use things like Puppeteer or Selenium. ...
Ethan Nichols's user avatar
0 votes
1 answer
76 views

Need an explanation for a web scraping lambda function in Python

I'm doing Web Scraping in Python and I've found this : products = soup.find_all('li') products_list = [] for product in products: name = product.h2.string price = product.find('p', string=...
Rayan CH TFG's user avatar
0 votes
0 answers
29 views

xpath to YouTube Subscribers is not working for me

I want to get the number of subscribers of a YouTube Channel into a Google Sheet without using the YouTube API. I want to export youtube subscribers count in Google Sheets. In a Google Sheet I tried ...
Rahul Raheja's user avatar
1 vote
2 answers
60 views

Getting correct selector using Puppeteer

here's the HTML code. <div class="list-row"> <div class="list-item"> <div class="imgframe"> <div class="img-wrap"&...
Oli Ver's user avatar
  • 21
1 vote
1 answer
40 views

Web scraping with Python BeautifulSoup

I would like to collect data by web scraping with Python BeautifulSoup from the website for my data analysis project. the data I want to collect from the website; date: 06.07.2027 Stage: Berghain, ...
hyoni's user avatar
  • 21
0 votes
1 answer
67 views

Scraping data from dropdown menu with no URL

I'm attempting to automate the process of downloading data for each US county from The Climate Explorer in R. The data I'm interested require the user to select from a dropdown menu, but I can't ...
user126004's user avatar
1 vote
1 answer
31 views

How do I use C# HtmlAgilityPack to get out the following text from a website?

I need to get out the text "223M" from the structure below using HtmlAgilityPack in C#, how do I do it? <div data-widget="quote-options-flow-summary" data-mode="desktop&...
Yu Deng's user avatar
  • 13
-1 votes
0 answers
43 views

Can I extract a database from a php website? [closed]

There is a website https://strongmanarchives.com/athletes.php that (if you click on the link) you will see is basically a database on a webpage, I wish to download the data so that I can run code on ...
George Bland's user avatar
0 votes
1 answer
51 views

Error when requesting via aio http AttributeError: the 'NoneType' object does not have the 'connect' attribute

I am writing an asynchronous parser http://freelance.habr.com where I iterate through all the tasks for a specific query. I use aio http for requests. During the execution, only some tasks give an ...
JFMAN12's user avatar
0 votes
0 answers
22 views

Undetected driver breaks down after third open url

I am doing some web scraping and I had to start using undetected chromedriver. The problem is that my code used to work with the normal chromedriver, but when using the undetected one it doesn't even ...
Lana Ramšak's user avatar
0 votes
1 answer
57 views

One specific span tag is empty with no attributes when using Selenium/SeleniumBase

I'm attempting to scrape data of baseball prospects on MLB.com with Python. When I open a browser manually and view the source HTML, I see the JSON data with everything I need in the single attribute ...
bdr's user avatar
  • 3
0 votes
0 answers
25 views

LinkedIn and Twitter Not Scraping My Next.js Page

I'm facing a bit of a challenge with getting LinkedIn and Twitter to properly scrape and preview one of my pages built with Next.js. Here’s what’s happening: The Issue: LinkedIn Post Inspector: When ...
Ren's user avatar
  • 101
0 votes
0 answers
29 views

scrape link for jwplayer calculated with JS using python

I'm trying to scrape video link (m3u8) from this website: https://deaddrive.xyz/embed/fa31e While inspecting the page, I realized that the link is calculated on the fly using JS in the function: &...
Gaurav Suman's user avatar
2 votes
2 answers
237 views

I can't get Selenium Chrome to work in Docker with Python

I have a classic "it works on my machine" problem, a web scraper I ran successfully on my laptop, but with a persistent error whenever I tried and run it in a container. My minimal ...
Max Power's user avatar
  • 8,652
1 vote
3 answers
70 views

403 error when trying to web scrape online job sites using python

I am trying to web scrape online job offering web sites for my coursera project. I keep getting a 403 error which, after I searched for its meaning online, I found out that it means that the web site ...
RockNRoll's user avatar
0 votes
0 answers
57 views

Recaptcha is mandatory error when try to bypass a recaptcha

I am trying to login the following site: https://agendamentos.mne.gov.pt/en/login using 2captcha and I implemented the codes correctly. The captcha is solved and sent to me, but when I click on the ...
Abdolbasir Tibash's user avatar
0 votes
0 answers
12 views

How to resolve Goose3 connection error 10054?

I run a script on daily basis using a cronjob and here we are using g.extract to extract articles and my script fails 2 out of 5 times with an error : Error occurred: ('Connection aborted.', ...
Gopal Goyal's user avatar
0 votes
1 answer
25 views

Web scraping using Apify with load more button

I am trying to scrape news articles from the website here using Apify-Smart Article Extractor. However, the webpage contains a 'load more' button which needs to be clicked in order to reveal required ...
marie20's user avatar
  • 773

1
2 3 4 5
1028