Questions tagged [screen-scraping]
Screen-scraping, also known as web-scraping or data-scraping, is a software technique used to collect and parse information from user interfaces. If your question is specifically about scraping from websites or web-APIs, please use the [web-scraping] tag instead.
4,176
questions
0
votes
0
answers
13
views
How to find the host from a web pages? [closed]
I'm trying to get an average of the most used hosts. So to get the hosts of web pages, until now all i found to do, is by looking at the request header when you "inspect" on the page and ...
-1
votes
0
answers
12
views
Using requests.post() for scraping data
So I'm trying to scrape birthdate data from the national voters websitetext. the problem is that the site generates some code in the payload that isn't gained from the source code(It's this ces_code). ...
-2
votes
1
answer
20
views
What's the best way to get just the user readble word content of a page? [closed]
Take your average mainstream url like:
https://people.com/books/jay-shetty-announces-new-book-8-rules-of-love/
And it's pretty easy to just copy and paste the text of the article as a human. But is ...
0
votes
0
answers
15
views
Cannot find reference 'Person' in 'linkedin_scraper.py'
Cannot find reference 'Person' in 'linkedin_scraper.py'
import os
from linkedin_scraper import Person, actions
from selenium import webdriver
driver = webdriver.Chrome("./chromedriver")
...
0
votes
0
answers
12
views
How can I Get value in tag with PHP simple HTML DOM parser
How can I to read data in tag url in the ?
I find in the manual and example but not found.
the tag like this in the below.
<tr role="cell" show-url="https://test.com"> </...
-1
votes
0
answers
33
views
Trying to extract headlines from a website using BeautifulSoup and it isn't working
I'm trying to extract headlines from a website.
When I go on the main page, it works, but I don't have enough data on that page.
When I go to a different URL with a search for a generic word like &...
0
votes
2
answers
47
views
Getting Links from a Dynamic Webpage
I am trying to get all links from a JS-based dynamic webpage with a load more option. I think I trigger THE page to scroll down. But, I could not get all the links. It turns only 103 links. I need to ...
0
votes
0
answers
17
views
Code getting stuck loading first page in multiple url webscrape request
I've written a small code to practice getting elements from multiple websites and put them in a csv file on my desktop. However, every time I run the code it gets stuck on the first URL after having ...
0
votes
0
answers
6
views
Scraping data in consistency
I am scraping data from a website; however, the scores do not match with the fixtures.
Scrape match data for each team and store it in a list of DataFrames
consolidated = []
for team_link in ...
0
votes
1
answer
33
views
How to properly load website using selenium in python
I'm trying to use selenium to open this link using the code below, but it seems to open just a very raw form of the website, with which I'm not managing to interact properly with the buttons.
options =...
1
vote
2
answers
46
views
R Glassdoor Web Scraping
I have been tasked to collect Glassdoor Reviews for different hospitals and I am having difficulties extracting the Pros, Cons, Advice to management, Recommend, CEO Approval, Business Outlook, and the ...
1
vote
1
answer
91
views
How to get data from table using "msxml2.xmlhttp" [duplicate]
HTMLI am trying to get data from a webpage https://www.lbma.org.uk/prices-and-data/precious-metal-prices#/table
Seemed simple with all the q and a examples online but I'm flapping around like a kipper ...
0
votes
2
answers
50
views
using python to scrape data from a tag that comes after a specific text on a website
So, I'm scraping some data from a website that was built strangely,
I have this part of the code to scrap the data:
# Find the span that contains the English name
try:
english_name_tag =...
0
votes
0
answers
44
views
Web Scraping Easy Apply Button From LinkedIn URL
I am looking for a tool or a script that will scrape a Linkedin URL job posting in Column D and Return a result of Easy Apply or Apply in Column E. I have a list of 1,000+ links and I want to know ...
0
votes
1
answer
38
views
why does the scraping from a specific website not work well?
I want to scrape all the alertDetails from this url - https://www.oref.org.il/12481-en/Pakar.aspx
I wrote this code but I think beacause the "In the past week" isn't Active, it can't find ...
-2
votes
0
answers
26
views
Scraping data from a document that has an API
I am a total newbie in JS and its frameworks, and I am trying to create a scraping bot that consumes data from documents generated in pandadoc (using a private key of course) and store that data ...
1
vote
2
answers
49
views
How to remove a tag when scraping
<p class="hidden-xs text-muted">
<b>GMC:</b>
123456
</p>
I simply need the GMC number from here which will appear as GMC : 123456 when I scrape it.
How to remove the ...
0
votes
0
answers
21
views
How do scraping in javascript websites and form required routes?
Crawling a html page is a relatively simple task: the crawler needs to identify the "a" tags and follow the href attributes, and there are several tools that do this.
But, what about when ...
0
votes
0
answers
17
views
Ebay Compatibility grabbing with php
I have had this script to parse Compatibility from ebay, for example
1 - getting content of page - https://www.ebay.co.uk/itm/173536511561
2 - parse "Bearer"
REGEX: '@{"value":&...
0
votes
0
answers
31
views
HTTPX returns 403, but requests 200. Python
I'm trying to parse one site. I'm sending a naked http request, just by setting a User-Agent.
It's surprising how a request using "requests" returns a 200 response. But the same query using ...
0
votes
0
answers
17
views
scraping only one side of job page on linkedin? using selenium
from selenium import webdriver
import time
import pandas as pd
import os
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium....
0
votes
0
answers
32
views
Web scraping react components which are rendered only when in view [duplicate]
I will like to scrape a React website, but the components are added to the DOM only when they are in view.
Is there a way to get through this?
1
vote
0
answers
219
views
A request made in Go lang gets blocked when Python requests and Curl works
I am making web scraper with golang, with the http package. I sent a GET request but it gets blocked by Cloudflare. I've wanted to port my tools over to Go but every time I try to, I get stuck at ...
0
votes
0
answers
90
views
Infinite scroll not working in headless mode with Python, Selenium and Chromedriver
I have the following script, which works ok to to scroll to the bottom of the page. It works when I don't use the --headless option. However, the infinite scroll does not work once I have introduced ...
1
vote
1
answer
30
views
Issue scraping dynamic table data from Human Microbiom Project (HMP) using python beautiful soup and selenium
I am working to web scrape the dynamic table data from the 'File UUID' column from the HMP website using python (beautiful soup and selenium). For some reason, I am able to pull all the data from the ...
1
vote
0
answers
29
views
How to Resume Scrapy Spider from the Last Known State if it Quits Due to Unexpected Reasons?
I am using Scrapy to scrape some big websites. Sometimes, my Scrapy spider quits due to unexpected reasons, and when I start it again, it runs from the start. This causes incomplete scraping of big ...
0
votes
0
answers
50
views
playwright handle JS initiated request and retrieve the response for scraping
The overall goal is to use Playwright for Python, to scrape a web page.
However, web page has some dynamically rendered info, like the Google Map. And I need to be able to get the longitude and ...
0
votes
0
answers
24
views
Want to import these images in to google sheet
I want to import these images from this category from this website https://www.daraz.pk/slicers-dicers/?from=onesearch_category_10000339&item_id=240649787&page=1&spm=a2a0e....
0
votes
0
answers
37
views
How do I get Control Properties from an AutomationElement
First let me state that I am currently working as an apprentice at a library resource provider. The software I'm primarily working on developing read data from tags and a keyboard emulator output ID ...
0
votes
0
answers
63
views
I'm creating a Python web scraping program using Selenium, but I encountered the following error: Stacktrace
I'm making a program that scrape photo on google with Selenium.
But I have a problems.
My code is here.
# Import modules
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
...
0
votes
0
answers
22
views
Scraping from interactive chart
I have been trying to scrape the data of an interactive chart on Morningstar using Selinium in Python. However, I am unable to find the data in the HTML source code and cannot seem to make other ...
1
vote
0
answers
36
views
BeautifulSoup returning different HTML for the pages that seems to have same HTML structure
I'm trying to scrape customer reviews from Amazon.com.
The thing is, BeautifulSoup is returning different HTML for different pages.
Because of this, I could only collect the reviews from the first ...
0
votes
1
answer
30
views
How to select a certain element in a div with selenium
I am scrapping a page, I need to select a certain element but there is multiple of them, this is how it is shown:
<div data-v-513c5331="" class="flex text-xs border-b h-9">......
0
votes
3
answers
118
views
How to scrape a dynamically loaded table inside a webpage?
I want to scrape the data from the table that is dynamically loaded inside this website.
I have tried the requests and selenium package in Python but seems like I have not implemented the right way.
...
0
votes
0
answers
25
views
How to built a website similar to restaurantguru.com in wordpress, with a pages for each restaurant using data from Google?
So I want to built a website similar to restaurantguru.com and others, as in an overview of restaurants, except only in Denmark, so not as big. I know how to use WordPress, but I'm a noob with API/...
1
vote
0
answers
63
views
Reuse chrome instance with Python/Selenium
I've found some information but not exactly what I need from google.
We have noticed that with our script it's opening ~9 chrome instances during running. We'd like it to just use one and then go onto ...
0
votes
0
answers
41
views
Copying entire line of source code in Google Chrome
I am trying to copy a whole line of source code, which is very long and so spreads across thousands of rows. Is there a shortcut to copying the whole line (either using python or just on chrome)? The ...
1
vote
2
answers
415
views
Cloudflare bypass python
I'm trying to parse a site with cloudflare protection.
On most cloudflare sites I've worked with, it would be sufficient to extract the cf_clearance cookie from the browser and paste it into the ...
0
votes
0
answers
37
views
Python program to scrap from a website the kanjis readings from a input (Ruby html)
My wordpress website theme wont leave the readings above the kanjis, so i thought i could create a python program that would return me the ruby html of the readings (furigana) of my kanjis input.
The ...
1
vote
0
answers
58
views
Extract element from a dropdown list
Good evening from Austria!
I have tried to access an element from a dropdown list, but without any success. The element where the dropdown list is stored is:
. Once the button to the dropdown has ...
1
vote
1
answer
25
views
data repetition in webscraping using xpath
I have a problem when im trying scraping data, when im looking an specific information
in google chrome browser console, this repeat it seven times and it goes to the next, here is my code
$x('//div[@...
0
votes
0
answers
18
views
Web Scraping Multiple Pages in Python 3 with unchanging URL
I'm fairly new at using Python and would like to scrape https://www.osac.gov/Content/Browse/Report?subContentTypes=Country%20Security%20Report , in particular the ContentItemId strings from the ...
0
votes
0
answers
33
views
Python Indeed scraping script does not report Company names
first time on python and I'm trying to scrape Indeed (IT market) to obtain infos about jobs posted by companies in certain combos "position+city".
I'm using Scrapy to do so, and it works ...
0
votes
1
answer
43
views
Pyhton Selenium No such Elements Everytings corrrect
I want to scrape the match results on this website using Python and Selenium. The results are stored in the tr tags within a table, but I've tried several methods and haven't been able to locate the ...
3
votes
1
answer
396
views
How to get a screenshot of a specific window with Rust on Windows?
I would like to know how to take screenshots of a part of a specific window. There may be an overlay on top of the application (a game overlay) that hides what I'm interested in. I would like to find ...
-1
votes
1
answer
26
views
telguarder.com/ scraping with Python
I am trying to scrape telguarder.com for DID status result located in specific CSS selector. And, exactly that selector returns such error. I have tried other selector from the same web page and it ...
-1
votes
3
answers
326
views
Capture Web Page screenshots at intervals with Puppeteer
I have a Puppeteer function that plays back the loading of a site by taking screenshots at a set interval:
const getScreenshots = async (browser, url, ms, frames): Promise<string[]> => {
...
0
votes
1
answer
63
views
Python requests, Loading in AJAX content
I'm trying to scrape all the 'AVAILABLE VACANCIES' from the URL https://careers.sega.co.uk/vacancies?f%5B0%5D=country%3AUnited%20Kingdom
I've written the following code:
import requests
def SEGA():
...
1
vote
0
answers
56
views
Data scraping and I get this problem: We're sorry but viewer-app doesn't work properly without JavaScript enabled. Please enable it to continue
I am attempting to scrape https://coworking.routesgrow.com/ .
When attempting to scrape with beautiful soup and normal requests library it wouldn't work. I switched to using selenium and beautiful ...
0
votes
1
answer
152
views
Unable to retrieve new html code after clicking next button using selenium. URL doesn't change as well
I am scraping https://www.coworker.com/search/turkey/izmir using selenium and beautiful soup. The html is rendered using Javascript which is why I am also using selenium. When clicking on the next ...