Questions tagged [screen-scraping]

Screen-scraping, also known as web-scraping or data-scraping, is a software technique used to collect and parse information from user interfaces. If your question is specifically about scraping from websites or web-APIs, please use the [web-scraping] tag instead.

screen-scraping
Filter by
Sorted by
Tagged with
1 vote
2 answers
27 views

Why is my status code 403 when trying to Scrape from GlassDoor?

I am trying to scrape job info from glass door. Similar code structure worked fine for LinkedIn, but I'm having issue here. I'm getting a Status Code of 403 so I'm guessing I'm running into some sort ...
0 votes
1 answer
41 views

How to resolve an HTTP error 503 reading Yahoo Finance from R?

The following code rvest::read_html("https://finance.yahoo.com/quote/VWIUX") generates Error in open.connection(x, "rb") : HTTP error 503. I know the url is ok because pasting ...
0 votes
2 answers
48 views

Get trimmed down text from Powershell

I am scraping version information from a website. I am able to get the information, but unable to get it without formatting. Currently targeting the DIV tag with Id j_idt19. Is there a way to get the ...
-3 votes
2 answers
52 views

Scraping presidential speeches with Selenium

I have been scraping with BS4 but never used Selenium before. Now I think I do. I created a list of links to old presidential speeches in the government's website - this is an example. (That's public ...
0 votes
1 answer
23 views

Capturing a Maximized Window in Secondary Screen Screenshot with Selenium

I'm a newbie using Selenium. I need to do a secondary screen screenshot of what's appearing there. I need the window to be maximized and appear on the screen 2. Is that possible? Can you please show ...
-1 votes
2 answers
57 views

Amazon Product Lookup Sales Data using SellerMagnet Scraping API Json Error

im new into coding and im starting off with my first project, i was trying to write an amazon script which will be able to scrape asin. Im using the scraping api from sellermagnet which is offering an ...
0 votes
2 answers
43 views

Extract the review ratings from i-herb webpage using python

I am facing a problem in trying to extract the ratings in the following webpage. https://gr.iherb.com/r/panoxyl-acne-foaming-wash-10-benzoyl-peroxide-maximum-strength-5-5-oz-156-g/103844 When I try to ...
0 votes
1 answer
27 views

Scraping ecoomerce website built with php

webiste is php on windows server tried everything from fetching html and text value but i need automation to format these values according to my requirements like product resolutio and all features *...
0 votes
1 answer
34 views

Scraping data from a refreshing javascript page in Python

I try to scrape some data from the site https://bloks.io/live The first problem is that I have no way to access the refreshing data in that table. My idea is to check if the first column changes. If ...
0 votes
1 answer
22 views

pagination, next page with scrapy

the next page button doesn't change the url when it's pressed, so i have problem with scrapy. ''' import scrapy class LegonSpider(scrapy.Spider): name = "legon" def start_requests(...
0 votes
1 answer
42 views

Simple ruby request using httparty with input parameters

I am learning ruby and attempting a simple ruby request on command line to scrape a website. There are 2 input elements with ids = "tb_radius_miles" and "locationSearchTextBox" I ...
-1 votes
1 answer
25 views

Web Scraping on Multiple Layers of a Website

I have a question regarding web scraping on multiple layers of a website. For instance, I have a website about US elections having 2 layers. Layer 1: state information: include 50 states. Once I click ...
11 votes
2 answers
28k views

Selenium Scroll inside of popup div

I am using selenium and trying to scroll inside the popup div on instagram. I get to a page like 'https://www.instagram.com/kimkardashian/', click followers, and then I can't get the followers list ...
6 votes
1 answer
3k views

Sending form data to aspx page

There is a need to do a search on the website url = r'http://www.cpso.on.ca/docsearch/' this is an aspx page (I'm beginning this trek as of yesterday, sorry for noob questions) using ...
4 votes
1 answer
2k views

Scroll down google reviews with selenium

I'm trying to scrape the reviews from this link: https://www.google.com/search?q=google+reviews+2nd+chance+treatment+40th+street&rlz=1C1JZAP_enUS697US697&oq=google+reviews+2nd+chance+...
0 votes
3 answers
2k views

Unable to scrape this site. How to scrape data from this site? [closed]

Iam not able to scrape data from this site. I tried with other sites but it's ok with other sites... from bs4 import BeautifulSoup from urllib.request import urlopen response = urlopen("https://www....
0 votes
0 answers
26 views

Why Excel PowerQuery result does not display all of the webpage content?

I am screen-scraping a web-page like this one (https://www.bmo.com/main/personal/bank-accounts/savings/interest-rates/) using Excel PowerQuery. The output contains only text fields, and not the ...
0 votes
0 answers
61 views

Puppeteer.Connect browserWSEndpoint with headless true?

The script below needs to connect to an already open instance of Google Chrome because I am already logged in there with my username and password. If it opens a new one, it will need to log in again, ...
6 votes
6 answers
2k views

Can I programmatically log into a website without storing the password in plaintext?

I do a number of projects that involve automatically submitting forms and/or retrieving data from websites. Some of these sites require username/password authentication. (These sites do not have APIs, ...
0 votes
0 answers
252 views

Web Scraping Easy Apply Button From LinkedIn URL

I am looking for a tool or a script that will scrape a Linkedin URL job posting in Column D and Return a result of Easy Apply or Apply in Column E. I have a list of 1,000+ links and I want to know ...
114 votes
2 answers
170k views

What's the best way of scraping data from a web site? [closed]

I need to extract contents from a web site, but the application doesn’t provide any application programming interface or another mechanism to access that data programmatically. I found a useful third-...
1 vote
2 answers
768 views

Cannot simulate this browser request using curl

I'm trying to scrape products from a site (e.g. https://www.violetgrey.com/en-us/shopping/the-rich-cream-18105401). Whilst on browser it loads normally, when I copy the initial curl request for the ...
0 votes
0 answers
27 views

Google Script UrlFetchApp.fetch() does not retrieve the entire HTML

I would like to extract the price of a product from this page link : https://gourmetmarketthailand.com/en/ritter_sport_almond_chocolate_100g_55161315168523 I am using UrlFetchApp.fetch() from Google ...
0 votes
1 answer
44 views

Why web-page in browser close so fast?

Sorry for my poor English When I try to use Pool for open several window browser -it's closed momently. I supposed it's happened course I got this error (I started to get this error recently, when I ...
0 votes
1 answer
95 views

Problems with reading a fixed width file?

I'm trying to run this simple script: from ibgeparser.microdados import Microdados from ibgeparser.enums import Anos, Estados, Modalidades if __name__ == "__main__": ano = Anos.DEZ ...
0 votes
0 answers
41 views

Way to periodically export data from an active web page to a text file or sheet?

I'm looking for a way to automatically regularly export data from a certain field of a website to a csv or text file from a browser like Chrome or Firefox, I've tried a lot of different extensions ...
7 votes
5 answers
3k views

Secure Communication Between iPhone and Server?

I'm developing an app which connects to an XML based API. I have control over both the server and the app - is there any way I can make sure only my app can access the API? There is no user ...
0 votes
0 answers
69 views

HAScript: How do you assign an AS400 screen field value to a variable (via Variable Update Action)?

From the guide, I must enter the name of the variable to be updated ($ScreenName$) into the 'Name' section, and the location string ('2,40') into the Value section. No matter what I try, the '...
0 votes
0 answers
57 views

Using python and visual studio, how do I intercept a toast notification on Windows, copy the text from it, and then paste it in desired location?

I am trying to create a program (an automated process) that logs into a certain website, which prompts a Duo Push (authentication method) that sends a text message to my phone. I need to copy this ...
0 votes
0 answers
18 views

How to use scrapy and selenium to interact with the body of the scrapy response?

I've seen a lot of people making a first request with the get method of the selenium webdriver and then scrape the response with scrapy. I want to do the opposite: a first request with scrapy and then ...
1 vote
1 answer
67 views

Scraping In R creates 18 tibbles

I am trying to learn how to scrape data in R. Using some help from other resources and chat gpt I have code that will scrape a table of NAIA baseball stats, but it creates 18 tibbles. It does grab all ...
0 votes
1 answer
35 views

How to get the hyperlink destinations for the results of this web search

I'm trying to write a Python script to scrape the hyperlink destinations (i.e., the href values) for each one of the entries in this web of science search results: https://www.webofscience.com/wos/...
0 votes
0 answers
19 views

Problem with action chains in interation using selenium and click()

i'm new to Python and i'm trying to build a automatic better in Roulette for training purpose. For now, i've a good setup and everything is fine. I'm passing a "result" object to my function ...
12 votes
2 answers
467 views

API/Data Sources - Linking and Pay-Per-Use [closed]

I am an independent developer who wanted to write a simple commercial app to display movie showtime information. After doing a whole lot of research, I could not find any good "legal" way of ...
1 vote
0 answers
48 views

Empty Json File when Web Scraping

I'm trying to scrape the website adamchoi for specific information using Scrapy and I'm getting a json file which is empty. All the code seems to be correct so this does not make sense. I've searching ...
24 votes
11 answers
33k views

Puppeteer waitForSelector on multiple selectors

I have Puppeteer controlling a website with a lookup form that can either return a result or a "No records found" message. How can I tell which was returned? waitForSelector seems to wait for only ...
0 votes
2 answers
3k views

Cloudflare bypass python

I'm trying to parse a site with cloudflare protection. On most cloudflare sites I've worked with, it would be sufficient to extract the cf_clearance cookie from the browser and paste it into the ...
0 votes
1 answer
60 views

change the language requests_html python

I have one site trying to scarp it using requests_html but it only take Arabic language I need the English text of the title Etc.. import pandas from requests_html import HTMLSession import time ...
0 votes
1 answer
68 views

How to find svg element using selenium and retrieve the value of date/time in a published Facebook post?

As you know recently Facebook is using svg tag to load data and time of published posts. I want to retrieve the value of some particular posts. The challenge is first do Hover and be wait till the ...
-2 votes
1 answer
215 views

Web Scraping Automation programs [closed]

This may be off topic but if anyone can help or point me in the right direction.....or the correct place to put this question? Can anyone suggest a good Automated Web Scraping program for use on a ...
0 votes
0 answers
77 views

Python Selenium Linux Headless Issue

I have been stuck for many days with a problem that does not allow me to advance my project. I'm trying to scrape onlyfans (obviously information that is available for free, such as profile photo, ...
0 votes
0 answers
121 views

What are the possibilities for reverse engineering GraphQL API to batch fetch information?

I’m working on a hobby project that revolves around Reddit’s avatar NFTs. Our small community of weirdos is passionate about collecting and trading these things. As Reddit generally doesn't care to ...
-1 votes
2 answers
1k views

Having problems with a simple Instagram Scraper [duplicate]

i am pretty new to all the programming stuff and I am learning Python for my social engineering project. So really sorry if you will hit your own forehead. So now i was looking at a tutorial to scrape ...
0 votes
0 answers
34 views

How to Download videos from Windows App (MS store app) for Offline Viewing?

Need help geeks, I have a subscription to an academy providing online video lectures for my course. The videos can only be accessed through a Microsoft Store based Windows application (probably a UWP ...
0 votes
0 answers
32 views

Tweet scraping--Error in twInterfaceObj$doAPICall(cmd, params, "GET", ...) : Forbidden (HTTP 403)

searchTwitter('#Houston', n=1000, lang='en')--after using this code, I am getting the below error-- Error in twInterfaceObj$doAPICall(cmd, params, "GET", ...) : Forbidden (HTTP 403). Is this ...
1 vote
0 answers
44 views

Selenium fails with page crash after adding login/passwd to simple script

I am using Python but I am not familiar with Selenium. My heat pump producer has a login page where I can see a lot of technical details about my heat pump, and I want to fetch these using a script ...
0 votes
0 answers
278 views

Amazon URL scraping with XMLHTTP - Amazon blocking?

So if I use my browser to browse to a list of a product's ellers, for example: https://www.amazon.co.uk/gp/offer-listing/B076C6769Z/ref=dp_olp_new?ie=UTF8&condition=new I see a list of sellers. ...
1 vote
1 answer
16 views

Web scraping from chronotrack running results with python

I'm trying to scrape results from a platform the running events results chronotrack. As an example, you can see the results for Santiago de Chile marathon 2022 on this link: https://results....
1 vote
1 answer
87 views

How do I scrape the text that appears when I mouse-hover the element?

On website https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i9-11900K+%40+3.50GHz&id=3904 I tried to scrape all the tool-tip information, the price and date of CPU in "Pricing history"...
13 votes
3 answers
21k views

BeautifulSoup and ASP.NET/C#

Has anyone integrated BeautifulSoup with ASP.NET/C# (possibly using IronPython or otherwise)? Is there a BeautifulSoup alternative or a port that works nicely with ASP.NET/C# The intent of planning ...

1
2 3 4 5
85