As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [screen-scraping]

Screen-scraping, also known as web-scraping or data-scraping, is a software technique used to collect and parse information from user interfaces. If your question is specifically about scraping from websites or web-APIs, please use the [web-scraping] tag instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

How to find the host from a web pages? [closed]

I'm trying to get an average of the most used hosts. So to get the hosts of web pages, until now all i found to do, is by looking at the request header when you "inspect" on the page and ...
Aayno's user avatar
  • 1
-1 votes
0 answers
12 views

Using requests.post() for scraping data

So I'm trying to scrape birthdate data from the national voters websitetext. the problem is that the site generates some code in the payload that isn't gained from the source code(It's this ces_code). ...
RIOMIO's user avatar
  • 1
-2 votes
1 answer
20 views

What's the best way to get just the user readble word content of a page? [closed]

Take your average mainstream url like: https://people.com/books/jay-shetty-announces-new-book-8-rules-of-love/ And it's pretty easy to just copy and paste the text of the article as a human. But is ...
Andrew Arrow's user avatar
  • 4,602
0 votes
0 answers
15 views

Cannot find reference 'Person' in 'linkedin_scraper.py'

Cannot find reference 'Person' in 'linkedin_scraper.py' import os from linkedin_scraper import Person, actions from selenium import webdriver driver = webdriver.Chrome("./chromedriver") ...
Bilal Khan's user avatar
0 votes
0 answers
12 views

How can I Get value in tag with PHP simple HTML DOM parser

How can I to read data in tag url in the ? I find in the manual and example but not found. the tag like this in the below. <tr role="cell" show-url="https://test.com"> </...
Thosabhorn Isarankura Na Ayudh's user avatar
-1 votes
0 answers
33 views

Trying to extract headlines from a website using BeautifulSoup and it isn't working

I'm trying to extract headlines from a website. When I go on the main page, it works, but I don't have enough data on that page. When I go to a different URL with a search for a generic word like &...
Keren Ben-yehuda's user avatar
0 votes
2 answers
47 views

Getting Links from a Dynamic Webpage

I am trying to get all links from a JS-based dynamic webpage with a load more option. I think I trigger THE page to scroll down. But, I could not get all the links. It turns only 103 links. I need to ...
kutsalzamazingo's user avatar
0 votes
0 answers
17 views

Code getting stuck loading first page in multiple url webscrape request

I've written a small code to practice getting elements from multiple websites and put them in a csv file on my desktop. However, every time I run the code it gets stuck on the first URL after having ...
kcast's user avatar
  • 1
0 votes
0 answers
6 views

Scraping data in consistency

I am scraping data from a website; however, the scores do not match with the fixtures. Scrape match data for each team and store it in a list of DataFrames consolidated = [] for team_link in ...
Bulelani Mxuma's user avatar
0 votes
1 answer
33 views

How to properly load website using selenium in python

I'm trying to use selenium to open this link using the code below, but it seems to open just a very raw form of the website, with which I'm not managing to interact properly with the buttons. options =...
tmbr's user avatar
  • 13
1 vote
2 answers
46 views

R Glassdoor Web Scraping

I have been tasked to collect Glassdoor Reviews for different hospitals and I am having difficulties extracting the Pros, Cons, Advice to management, Recommend, CEO Approval, Business Outlook, and the ...
AndrooGC's user avatar
1 vote
1 answer
91 views

How to get data from table using "msxml2.xmlhttp" [duplicate]

HTMLI am trying to get data from a webpage https://www.lbma.org.uk/prices-and-data/precious-metal-prices#/table Seemed simple with all the q and a examples online but I'm flapping around like a kipper ...
G Knowles's user avatar
0 votes
2 answers
50 views

using python to scrape data from a tag that comes after a specific text on a website

So, I'm scraping some data from a website that was built strangely, I have this part of the code to scrap the data: # Find the span that contains the English name try: english_name_tag =...
lidorkalfa's user avatar
0 votes
0 answers
44 views

Web Scraping Easy Apply Button From LinkedIn URL

I am looking for a tool or a script that will scrape a Linkedin URL job posting in Column D and Return a result of Easy Apply or Apply in Column E. I have a list of 1,000+ links and I want to know ...
Michael LaMarche's user avatar
0 votes
1 answer
38 views

why does the scraping from a specific website not work well?

I want to scrape all the alertDetails from this url - https://www.oref.org.il/12481-en/Pakar.aspx I wrote this code but I think beacause the "In the past week" isn't Active, it can't find ...
חי אלימלך's user avatar
-2 votes
0 answers
26 views

Scraping data from a document that has an API

I am a total newbie in JS and its frameworks, and I am trying to create a scraping bot that consumes data from documents generated in pandadoc (using a private key of course) and store that data ...
Jaime Sacristán's user avatar
1 vote
2 answers
49 views

How to remove a tag when scraping

<p class="hidden-xs text-muted"> <b>GMC:</b> 123456 </p> I simply need the GMC number from here which will appear as GMC : 123456 when I scrape it. How to remove the ...
Nikhil V's user avatar
0 votes
0 answers
21 views

How do scraping in javascript websites and form required routes?

Crawling a html page is a relatively simple task: the crawler needs to identify the "a" tags and follow the href attributes, and there are several tools that do this. But, what about when ...
Arivan Bastos's user avatar
0 votes
0 answers
17 views

Ebay Compatibility grabbing with php

I have had this script to parse Compatibility from ebay, for example 1 - getting content of page - https://www.ebay.co.uk/itm/173536511561 2 - parse "Bearer" REGEX: '@{"value":&...
Bookamba's user avatar
0 votes
0 answers
31 views

HTTPX returns 403, but requests 200. Python

I'm trying to parse one site. I'm sending a naked http request, just by setting a User-Agent. It's surprising how a request using "requests" returns a 200 response. But the same query using ...
Ivan's user avatar
  • 27
0 votes
0 answers
17 views

scraping only one side of job page on linkedin? using selenium

from selenium import webdriver import time import pandas as pd import os from selenium.webdriver.support.select import Select from selenium.webdriver.support.ui import WebDriverWait from selenium....
Muhammadyusuf Yo'ldashev's user avatar
0 votes
0 answers
32 views

Web scraping react components which are rendered only when in view [duplicate]

I will like to scrape a React website, but the components are added to the DOM only when they are in view. Is there a way to get through this?
Alb Rick Bi's user avatar
1 vote
0 answers
219 views

A request made in Go lang gets blocked when Python requests and Curl works

I am making web scraper with golang, with the http package. I sent a GET request but it gets blocked by Cloudflare. I've wanted to port my tools over to Go but every time I try to, I get stuck at ...
sweet's user avatar
  • 15
0 votes
0 answers
90 views

Infinite scroll not working in headless mode with Python, Selenium and Chromedriver

I have the following script, which works ok to to scroll to the bottom of the page. It works when I don't use the --headless option. However, the infinite scroll does not work once I have introduced ...
Tendekai Muchenje's user avatar
1 vote
1 answer
30 views

Issue scraping dynamic table data from Human Microbiom Project (HMP) using python beautiful soup and selenium

I am working to web scrape the dynamic table data from the 'File UUID' column from the HMP website using python (beautiful soup and selenium). For some reason, I am able to pull all the data from the ...
Nathaniel Landi's user avatar
1 vote
0 answers
29 views

How to Resume Scrapy Spider from the Last Known State if it Quits Due to Unexpected Reasons?

I am using Scrapy to scrape some big websites. Sometimes, my Scrapy spider quits due to unexpected reasons, and when I start it again, it runs from the start. This causes incomplete scraping of big ...
Waqas Arain's user avatar
0 votes
0 answers
50 views

playwright handle JS initiated request and retrieve the response for scraping

The overall goal is to use Playwright for Python, to scrape a web page. However, web page has some dynamically rendered info, like the Google Map. And I need to be able to get the longitude and ...
Ilija's user avatar
  • 1,546
0 votes
0 answers
24 views

Want to import these images in to google sheet

I want to import these images from this category from this website https://www.daraz.pk/slicers-dicers/?from=onesearch_category_10000339&item_id=240649787&page=1&spm=a2a0e....
M Basit Mehtab Awan's user avatar
0 votes
0 answers
37 views

How do I get Control Properties from an AutomationElement

First let me state that I am currently working as an apprentice at a library resource provider. The software I'm primarily working on developing read data from tags and a keyboard emulator output ID ...
Jonas Alexander Offersen's user avatar
0 votes
0 answers
63 views

I'm creating a Python web scraping program using Selenium, but I encountered the following error: Stacktrace

I'm making a program that scrape photo on google with Selenium. But I have a problems. My code is here. # Import modules import requests from bs4 import BeautifulSoup from selenium import webdriver ...
박동규's user avatar
0 votes
0 answers
22 views

Scraping from interactive chart

I have been trying to scrape the data of an interactive chart on Morningstar using Selinium in Python. However, I am unable to find the data in the HTML source code and cannot seem to make other ...
Karl Emil Thulstrup's user avatar
1 vote
0 answers
36 views

BeautifulSoup returning different HTML for the pages that seems to have same HTML structure

I'm trying to scrape customer reviews from Amazon.com. The thing is, BeautifulSoup is returning different HTML for different pages. Because of this, I could only collect the reviews from the first ...
canonlaw's user avatar
0 votes
1 answer
30 views

How to select a certain element in a div with selenium

I am scrapping a page, I need to select a certain element but there is multiple of them, this is how it is shown: <div data-v-513c5331="" class="flex text-xs border-b h-9">......
eric's user avatar
  • 1
0 votes
3 answers
118 views

How to scrape a dynamically loaded table inside a webpage?

I want to scrape the data from the table that is dynamically loaded inside this website. I have tried the requests and selenium package in Python but seems like I have not implemented the right way. ...
Khang Nguyen's user avatar
0 votes
0 answers
25 views

How to built a website similar to restaurantguru.com in wordpress, with a pages for each restaurant using data from Google?

So I want to built a website similar to restaurantguru.com and others, as in an overview of restaurants, except only in Denmark, so not as big. I know how to use WordPress, but I'm a noob with API/...
Marcus Kirkegaard's user avatar
1 vote
0 answers
63 views

Reuse chrome instance with Python/Selenium

I've found some information but not exactly what I need from google. We have noticed that with our script it's opening ~9 chrome instances during running. We'd like it to just use one and then go onto ...
user avatar
0 votes
0 answers
41 views

Copying entire line of source code in Google Chrome

I am trying to copy a whole line of source code, which is very long and so spreads across thousands of rows. Is there a shortcut to copying the whole line (either using python or just on chrome)? The ...
Ben303's user avatar
  • 35
1 vote
2 answers
415 views

Cloudflare bypass python

I'm trying to parse a site with cloudflare protection. On most cloudflare sites I've worked with, it would be sufficient to extract the cf_clearance cookie from the browser and paste it into the ...
Ivan's user avatar
  • 27
0 votes
0 answers
37 views

Python program to scrap from a website the kanjis readings from a input (Ruby html)

My wordpress website theme wont leave the readings above the kanjis, so i thought i could create a python program that would return me the ruby html of the readings (furigana) of my kanjis input. The ...
TPAL's user avatar
  • 3
1 vote
0 answers
58 views

Extract element from a dropdown list

Good evening from Austria! I have tried to access an element from a dropdown list, but without any success. The element where the dropdown list is stored is: . Once the button to the dropdown has ...
linus otte's user avatar
1 vote
1 answer
25 views

data repetition in webscraping using xpath

I have a problem when im trying scraping data, when im looking an specific information in google chrome browser console, this repeat it seven times and it goes to the next, here is my code $x('//div[@...
Benjamin Correa's user avatar
0 votes
0 answers
18 views

Web Scraping Multiple Pages in Python 3 with unchanging URL

I'm fairly new at using Python and would like to scrape https://www.osac.gov/Content/Browse/Report?subContentTypes=Country%20Security%20Report , in particular the ContentItemId strings from the ...
sleepypeepers's user avatar
0 votes
0 answers
33 views

Python Indeed scraping script does not report Company names

first time on python and I'm trying to scrape Indeed (IT market) to obtain infos about jobs posted by companies in certain combos "position+city". I'm using Scrapy to do so, and it works ...
argimboldo's user avatar
0 votes
1 answer
43 views

Pyhton Selenium No such Elements Everytings corrrect

I want to scrape the match results on this website using Python and Selenium. The results are stored in the tr tags within a table, but I've tried several methods and haven't been able to locate the ...
Sezo ka's user avatar
3 votes
1 answer
396 views

How to get a screenshot of a specific window with Rust on Windows?

I would like to know how to take screenshots of a part of a specific window. There may be an overlay on top of the application (a game overlay) that hides what I'm interested in. I would like to find ...
Unknow's user avatar
  • 168
-1 votes
1 answer
26 views

telguarder.com/ scraping with Python

I am trying to scrape telguarder.com for DID status result located in specific CSS selector. And, exactly that selector returns such error. I have tried other selector from the same web page and it ...
Arthur's user avatar
  • 1
-1 votes
3 answers
326 views

Capture Web Page screenshots at intervals with Puppeteer

I have a Puppeteer function that plays back the loading of a site by taking screenshots at a set interval: const getScreenshots = async (browser, url, ms, frames): Promise<string[]> => { ...
Harry's user avatar
  • 4,606
0 votes
1 answer
63 views

Python requests, Loading in AJAX content

I'm trying to scrape all the 'AVAILABLE VACANCIES' from the URL https://careers.sega.co.uk/vacancies?f%5B0%5D=country%3AUnited%20Kingdom I've written the following code: import requests def SEGA(): ...
scotmanDavid's user avatar
1 vote
0 answers
56 views

Data scraping and I get this problem: We're sorry but viewer-app doesn't work properly without JavaScript enabled. Please enable it to continue

I am attempting to scrape https://coworking.routesgrow.com/ . When attempting to scrape with beautiful soup and normal requests library it wouldn't work. I switched to using selenium and beautiful ...
Mohi Eldin  's user avatar
0 votes
1 answer
152 views

Unable to retrieve new html code after clicking next button using selenium. URL doesn't change as well

I am scraping https://www.coworker.com/search/turkey/izmir using selenium and beautiful soup. The html is rendered using Javascript which is why I am also using selenium. When clicking on the next ...
Mohi Eldin  's user avatar

1
2 3 4 5
84