Questions tagged [web-scraping]

Web scraping is the process of extracting specific information from websites that do not readily provide an API or other methods of automated data retrieval. Questions about "How To Get Started With Scraping" (e.g. with Excel VBA) should be *thoroughly researched* as numerous functional code samples are available. Web scraping methods include 3rd-party applications, development of custom software, or even manual data collection in a standardized way.

Filter by
Sorted by
Tagged with
0 votes
3 answers
15 views

web scraping wikipedia using selenium in python

from selenium import webdriver # Initialize the webdriver driver = webdriver.Edge() WIKI = "https://en.wikipedia.org/wiki/India" driver.get(WIKI) # Find the <div> element with class ...
Keshav Verma's user avatar
0 votes
1 answer
39 views

Scrapy - xpath returns empty list

I'm scraping restaurant reviews from yelp, specifically from this url I'm trying to get the list of review containers and, after testing with the chrome console, that would be given by the following ...
Maurizio Brini's user avatar
0 votes
1 answer
48 views

Can't click on button in shadow-dom

I'm trying to scrape a dynamic site (https://app.shaggyowl.com/accesso-titolare/index.html) with selenium and python. I found a workaround to insert username and password but I'm not able to click ...
Federico's user avatar
0 votes
1 answer
26 views

HtmlUnit: Handling Hidden Google reCAPTCHA Token in Login Automation

I'm using HtmlUnit to automate the login process on a website. The website employs a Google reCAPTCHA to protect its login form, and the reCAPTCHA token is hidden in the HTML. I need to obtain and use ...
Abraham Arnold's user avatar
1 vote
1 answer
31 views

Selenium Web Scraping table with "next" button

I'm looking to get the data from this open source url: url = "https://data.sfgov.org/Public-Safety/Map-of-Police-Department-Incident-Reports-2018-to-/jq29-s5wp" There is a table at the ...
ZahlenZ's user avatar
  • 13
0 votes
0 answers
53 views

selenium.common.exceptions.NoSuchDriverException: Message: Unable to obtain driver for chrome using Selenium Manager

I am writing bot using discord.py and selenium. Code runs normally on my computer, but when I try to deploy it on server, an NoSuchDriverException occurs. Before running code I've installed everything ...
Ivan's user avatar
  • 1
0 votes
1 answer
27 views

How to connect to a proxy server with puppeteer?

I am trying now for a while to connect to a proxy with puppeteer. I am using the latest version and I couldn't get it to connect after endless tries. I searched, I documented, I read but I don't even ...
Flechas's user avatar
-2 votes
0 answers
23 views

Scraping Government Website for Compliance Data: Handling Missing Data and Website Issues [closed]

I am scraping compliance data from a government website using Python, and I've encountered issues when there is no data for certain years or some pages fail to load, possibly due to website problems. ...
kizzeee's user avatar
0 votes
0 answers
37 views

i can't open twitter via google chrome web driver (I have a token) Python Chrome 117 version error

from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.chrome.options import Options from selenium.common.exceptions import NoSuchElementException from ...
Latte cofie's user avatar
-1 votes
0 answers
53 views

VBA code showing error Object variable not set (Error 91)

1st Error : Dim myName22 As String myName22 = Sheet6.Range("G2").Value **doc.getElementsByName("mainForm:j_id_1t_d")(0).Value = myName22** Getting error "Object variable ...
Rupesh Ghatak's user avatar
0 votes
2 answers
48 views

webscrape and export csv

I need to webscrape and I don't know what to put for the userdetail[].text or if that's even what I should be using. I'm also getting AttributeError: 'NoneType' object has no attribute 'find_all' ...
dee's user avatar
  • 1
-2 votes
0 answers
24 views

How can I scrappe 100,000 URL profiles [closed]

I have a list of 100,000 url of a website, I already detected all element class (how to select them) no what is the libraries that can acheive the fastest way to scrappe all that ? I already used ...
Mohcine221's user avatar
0 votes
0 answers
11 views

scarping cp website codes error with request

i am trying to scrape a cp-website but due to lot of requests i made to the site it blocks my IP so i am trying to achieve so by making a rotatory proxy so i get some free proxies and i am checking ...
Mostafa M. Galal's user avatar
-2 votes
0 answers
22 views

Web scrapping Nodejs application [closed]

Please help me people who know. I want to create a node js application to extract the url of the videos played on websites like streamsb, streamtape etc. What can be used to do this...? and how A ...
Oshan Avishka's user avatar
0 votes
1 answer
26 views

Why is there an encoding error when scraping aliexpess in python?

I am trying to sparse data about users who have left product reviews, here is my code: from requests import Session prod_link = 'https://aliexpress.ru/aer-jsonapi/review/v2/desktop/product/reviews? ...
kuchuluk's user avatar
0 votes
0 answers
16 views

Get method payments of e-commerce websites [closed]

I have a list of over 20 000 e-commerce websites and i need to get the payment methods for each. Which technology would you recommend and how to do it exactly? The websites are very different from ...
Jenn's user avatar
  • 1
0 votes
3 answers
49 views

Scraping with Beautifulsoup: Not all class values returned

I am trying to scrape a few values for a website with Beautifulsoup. I am able to identify the class and export those. Unfortunately, the crucial values are missing. Would anyone here be able to help ...
David's user avatar
  • 11
0 votes
1 answer
33 views

HTTP 403 and 308 status code in Python3.10 Web Scratch

I try to scrape a website for homework, with firstly a Error 403 Forbidden returned, the code is: from urllib.request import urlopen, Request from bs4 import BeautifulSoup url = "https://www....
Zixuan Ling's user avatar
2 votes
1 answer
23 views

Python live script not functioning to check for new emails in gmail

I am trying to build an application script to parse through new incoming emails I have check my script on already existing emails from specific sender and it runs fine but since I have modified it ...
Raafeh Sajjad's user avatar
0 votes
0 answers
19 views

Does login with Instagram use session ID still exist?

Currently, I'm trying to learn Puppeteer on Node.js to access an Instagram account page without using a username and password for login. I believe that if I continuously attempt to log in with my ...
ferdinand's user avatar
  • 325
-1 votes
0 answers
29 views

How to solve Scrapy not recognizing [] at the class name?

I'm new to webscraping. Making my first data collection. I'm trying to extract the text from a tag using scrapy in Powershell. I`d run the following code response.css('span.overflow-hidden text-...
TayLo's user avatar
  • 1
0 votes
0 answers
30 views

I am trying to scrape images from a website with playwright. I am unable to scrape its one div content load dynamically

I totally don't get the page load normally, still, bigImage is empty. Here's the below code. I am using asyncio and playwright along with lxlm and requests. I am trying to wait for the element to load ...
Atul Anand Oraon's user avatar
0 votes
1 answer
30 views

error: can't login on Web of Science website with python selenium webscaper

I'm trying to login on Web of Science: https://mjl.clarivate.com/login;createAccount=false;referrer=%2Fhome I have to fill the "Email address" and "Password" boxes with my data, ...
pc_'s user avatar
  • 1
0 votes
0 answers
13 views

Can't recreate API call from CURL [closed]

Even though I copy and paste the curl to postman that works fine on site's network tab It does not works and returns captcha in Postman why is this happening? I am new to web scraping and APIs so I ...
bora aslan's user avatar
-1 votes
1 answer
25 views

BeautifulSoup scraping Div class element second argument

I'm trying to use BS for the web scrapping. I can easily find the defined class, see the screen below, but it will show me the all ements, with all sizes. Is there a way, how to filter the second ...
frky's user avatar
  • 5
-2 votes
1 answer
16 views

Web Scrapping issues while getting links [closed]

So, I'm doing a project where I have to get some csv files from an specific website. The issue is that I'm not getting all the links. I've done the process one by one (until the 3th-one), but when I ...
Angel Panda's user avatar
-4 votes
1 answer
36 views

Scraping Barchart.com financial data

I am trying to scrape the financial data in the tables on https://www.barchart.com/stocks/quotes/IBM/income-statement/ Using inspect element I didn't see any XHR/fetch requests, but it looks like the ...
Adam's user avatar
  • 425
-3 votes
0 answers
28 views

HTTP2 protocol ERROR when scraping Sealed SOTHEBY with Selenium (bug occurs 80% of the time)

I've written a script that goes through all the pages (that have listings of auctioned items), collecting all the items' url, then open each url and download the 1st image of each item. There are 2 ...
Nova's user avatar
  • 1
-1 votes
0 answers
18 views

Extract data from a given HTML with Selenium [closed]

I have an html text I took it from a website like that : <div id="client-focus-attributes-section" class="attributes details-section client-focus-attributes-section" data-v-...
Mohcine221's user avatar
1 vote
2 answers
42 views

Can't produce tabular content from a webpage using Selenium

After I click on the See all hours button located in the upper left corner next to the Closed now button on this webpage, a box containing a schedule appears. I've created a script in Selenium to ...
MITHU's user avatar
  • 139
-1 votes
0 answers
26 views

Python web scraper - problem with loading another page

So I'm trying to build a web scraper in python that scrapes a title, date and link of an article from a webpage, within a certain time range - and that part works. However, once it finishes scraping ...
annie.sturm's user avatar
0 votes
0 answers
21 views

Should I wait after searching for elements and/or scrolling while web scraping?

I'm trying to understand when to wait during a simple python/selenium web scraping script. I'm pulling titles and links for the top 100 results for approximately 50 search terms. I understand some ...
Jonathan Cakes's user avatar
0 votes
0 answers
31 views

how to scrape website with infinite scroll?

I want to scrape website in which, when we scroll down scrollbar the new data starts loading. how to scrape this website? hospitals2 = [] for i in range(1,10): try: element = driver....
Arpit 's user avatar
0 votes
3 answers
44 views

Web Scraping and searching on website via python

I am trying to webscrape power of 10, the website requires a first/last name and a club, i will be getting all this from python . I want to automatically search this website and enter the details, ...
dexta's user avatar
  • 25
0 votes
0 answers
24 views

Scraping these website but the url not change in payload [closed]

I scraped this website but it URL not changing in the payload also kindly enter k in the search field then they will show you records CODE import requests from bs4 import BeautifulSoup import json ...
Amen Aziz's user avatar
  • 161
-1 votes
0 answers
23 views

Python insta-scrape module is giving ValueError

I found a module called insta-scrape to download Instagram reels, but when I run the following code, I get a ValueError. This is the code I am running: from insta_scrape import Reel import time # ...
soul upgrader's user avatar
0 votes
1 answer
34 views

Error length(url) == 1 is not TRUE when web scraping

I have got the following link: https://www.senamhi.gob.pe/mapas/mapa-estaciones-2/_dato_esta_tipo02.php?estaciones=109045&CBOFiltro=202112&t_e=M&estado=DIFERIDO&cod_old=154107&...
César Carvallo's user avatar
0 votes
1 answer
45 views

Are there faster way to scrape data

I use for loop to scrape the table data. https://money.rediff.com/gainers values = driver.find_elements(By.XPATH,"//a[contains(text(), 'Rainbow Foundati')]/ancestor::tr/following-sibling::*")...
Alex's user avatar
  • 61
0 votes
0 answers
22 views

Is there a way to extract "event_type" from an event on Forex Factory from the "inspect source" page using R Studio?

I'm trying to extract what "event_type" an event is from Forex Factory's calendar page. For example if inspect the source code of the page "https://www.forexfactory.com/calendar?week=...
Nick Amato's user avatar
0 votes
1 answer
31 views

Scrapy - Only first url in url list is scraped

I'm scraping reviews from restaurants in Rome, Milan and Bergamo. For each one of those cities there's one dedicated url containing 30 or more restaurants. The scraper starts crawling the Rome ...
Maurizio Brini's user avatar
0 votes
1 answer
51 views

What wrong in Puppeteer technology, why most site show another view on directly browsing and browsing with Puppeteer?

I do checked a serious of sites directly and with Puppeteer with that code How can I capture all network requests and full response data when loading a page in Chrome? Starting from https://www....
Alex's user avatar
  • 387
0 votes
0 answers
9 views

How can I scrape google scholar using key words in R?

I'm trying to measure the variation in the number of publications for different species. There is an R package that does this called specieshindex but I'm having trouble getting it set up. I was ...
adkane's user avatar
  • 1,429
-3 votes
0 answers
25 views

Difficulty Tracking Hotel Prices Due to Currency Changes

I've been working on a Python web scraping project to monitor hotel prices, and I've run into a challenging issue. The prices on the hotel website sometimes change currency, for example, from ILS to ...
Elior's user avatar
  • 35
0 votes
1 answer
19 views

Getting the content specified under var using rvest package

I want to extract the following information from the link https://www.betashares.com.au/fund/high-interest-cash-etf/ I have written the following code: link <- "https://www.betashares.com.au/...
shani's user avatar
  • 217
0 votes
0 answers
10 views

cpcb site scraper giving unauthorized access in output

https://github.com/thejeshgn/cpcbccr, i was trying to use this get pollutant data from cpcb site, i was able to run setup_pull.py without any error but when i try to run pull.py it runs but give ...
Himanshu Shekhar's user avatar
0 votes
0 answers
43 views

Python to scrape a dynamic page

This is the page I'm attempting to scrape: https://developer.servicenow.com/dev.do#!/reference/api/vancouver/server/no-namespace/ActionAPIBoth I'm trying to get all the //*[@id="...
Lewis's user avatar
  • 135
1 vote
1 answer
41 views

How to scrape node's text which has <!-- --> inside?

I am writing a web crawler for scraping information from job boards. I completed my first crawler, but it has some more issues to resolve. For some companies titles I get this ПАО\xa0 as a result. ...
Gleichmut's user avatar
  • 6,013
0 votes
1 answer
49 views

WWW::Scripter package using 1.2GB of memory while being used for webscraping

I am not super familiar with this package in the first place. I discovered that the use_plugin('JavaScript') method consumes alot of memory through a profiler. I swapped this method for the plugin('...
Rizwan Ahamed's user avatar
0 votes
0 answers
15 views

How to Scraping by Tweepy

This is my first time using Tweepy to scraping my tweet, i use Google colab and i get stuck while make a query # Define query query = 'from:BarackObama -is:retweet' # get max. 100 tweets tweets = ...
dewcxiv's user avatar
-1 votes
1 answer
19 views

Running scrapy spider but blank output. python

I am trying to have this spider go through a list of 1600 urls contained in a csv and pull emails and phone numbers from the page. If anyone already has such a program I'd be happy to use it, but also ...
BegginerScraper Griff's user avatar

1
2 3 4 5
993