Questions tagged [screen-scraping]

Screen-scraping, also known as web-scraping or data-scraping, is a software technique used to collect and parse information from user interfaces. If your question is specifically about scraping from websites or web-APIs, please use the [web-scraping] tag instead.

screen-scraping
Filter by
Sorted by
Tagged with
10 votes
3 answers
6k views

OpenGL/D3D: How do I get a screen grab of a game running full screen in Windows?

Suppose I have an OpenGL game running full screen (Left 4 Dead 2). I'd like to programmatically get a screen grab of it and then write it to a video file. I've tried GDI, D3D, and OpenGL methods (eg ...
Theodore Baxley's user avatar
8 votes
2 answers
11k views

What's the best approach for parsing XML/'screen scraping' in iOS? UIWebview or NSXMLParser?

I am creating an iOS app that needs to get some data from a web page. My first though was to use NSXMLParser initWithContentsOfURL: and parse the HTML with the NSXMLParser delegate. However this ...
Benedict Cohen's user avatar
1 vote
1 answer
3k views

Webbrowser behaviour issues

I am trying to automate Webbrowser with .NET C#. The issue is that the control or should I say IE browser behaves strange on different computers. For example, I am clickin on link and fillup a Ajax ...
Jim's user avatar
  • 2,816
33 votes
8 answers
22k views

PHP CSS Selector Library? [closed]

Is there a PHP class/library that would allow me to query an XHTML document with CSS selectors? I need to scrape some pages for data that is very easily accessible if I could somehow use CSS selectors ...
Wilco's user avatar
  • 33k
18 votes
6 answers
29k views

Scraping websites with Javascript enabled?

I'm trying to scrape and submit information to websites that heavily rely on Javascript to do most of its actions. The website won't even work when i disable Javascript in my browser. I've searched ...
user216171's user avatar
  • 1,696
17 votes
3 answers
34k views

How can I read and parse the contents of a webpage in R

I'd like to read the contents of a URL (e.q., http://www.haaretz.com/) in R. I am wondering how I can do it
Mark's user avatar
  • 10.9k
11 votes
1 answer
41k views

Extracting table contents from html with python and BeautifulSoup

I want to extract certain information out of an html document. E.g. it contains a table (among other tables with other contents) like this: <table class="details"> <tr> ...
Isaac's user avatar
  • 810
7 votes
2 answers
11k views

Scraping javascript-generated data using Python

I want to scrape some data of following url using Python. http://www.hankyung.com/stockplus/main.php?module=stock&mode=stock_analysis_infomation&itemcode=078340 It's about a summary of ...
trigger's user avatar
  • 91
7 votes
3 answers
6k views

How to replace words with span tag using jsoup?

Assume I have the following html: <html> <head> </head> <body> <div id="wrapper" > <div class="s2">I am going <a title="some title" href="">by ...
rjc's user avatar
  • 2,905
3 votes
2 answers
2k views

Evaluate javascript on a local html file (without browser)

This is part of a project I am working on for work. I want to automate a Sharepoint site, specifically to pull data out of a database that I and my coworkers only have front-end access to. I FINALLY ...
araisbec's user avatar
  • 1,233
3 votes
4 answers
7k views

scrapy log handler

I seek your help in the following 2 questions - How do I set the handler for the different log levels like in python. Currently, I have STATS_ENABLED = True STATS_DUMP = True LOG_FILE = 'crawl.log'...
goh's user avatar
  • 28.7k
2 votes
2 answers
4k views

C# WebClient - View source question

I'm using a C# WebClient to post login details to a page and read the all the results. The page I am trying to load includes flash (which, in the browser, translates into HTML). I'm guessing it's ...
BIDeveloper's user avatar
  • 2,628
2 votes
1 answer
7k views

Why is contains(text(), "string" ) not working in XPath?

I have written this expression //*[contains(text(), "Brand:" )] for the below HTML code. <div class="info-product mt-3"> <h3>Informazioni prodotto</h3> Brand: <...
stack's user avatar
  • 49
2 votes
1 answer
5k views

CasperJS click event having AJAX call

I am trying to fetch data from a site by simulating events using CasperJS with phantomJS 1.7.0. I am able to simulate normal click events and select events. But my code fails in following scenario: ...
Learn More's user avatar
  • 1,561
1 vote
3 answers
2k views

Scraping YouTube links from a webpage

I've been trying to scrape YouTube links from a webpage, but nothing has worked. This is a picture of what I've been trying to scrape.: This is the code I tried most recently: youtube_link = soup....
adomzy's user avatar
  • 85
114 votes
2 answers
170k views

What's the best way of scraping data from a web site? [closed]

I need to extract contents from a web site, but the application doesn’t provide any application programming interface or another mechanism to access that data programmatically. I found a useful third-...
0x1ad2's user avatar
  • 8,034
84 votes
7 answers
80k views

How does a site like kayak.com aggregate content? [closed]

Greetings, I've been toying with an idea for a new project and was wondering if anyone has any idea on how a service like Kayak.com is able to aggregate data from so many sources so quickly and ...
Jeff's user avatar
  • 2,828
55 votes
8 answers
51k views

Screen scraping: getting around "HTTP Error 403: request disallowed by robots.txt"

Is there a way to get around the following? httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt Is the only way around this to contact the site-owner (barnesandnoble.com).. i'm ...
Diego's user avatar
  • 795
37 votes
3 answers
87k views

BeautifulSoup: How do I extract all the <li>s from a list of <ul>s that contains some nested <ul>s?

I'm a newbie programmer trying to jump in to Python by building a script that scrapes http://en.wikipedia.org/wiki/2000s_in_film and extracts a list of "Movie Title (Year)". My HTML source ...
danneu's user avatar
  • 9,344
31 votes
3 answers
75k views

XPath to Parse "SRC" from IMG tag?

Right now I successfully grabbed the full element from an HTML page with this: //img[@class='photo-large'] for example it would return this: <img src="http://example.com/img.jpg" class='photo-...
kush's user avatar
  • 16.7k
24 votes
11 answers
33k views

Puppeteer waitForSelector on multiple selectors

I have Puppeteer controlling a website with a lookup form that can either return a result or a "No records found" message. How can I tell which was returned? waitForSelector seems to wait for only ...
Jon Wilson's user avatar
23 votes
2 answers
22k views

Scrapy, scraping data inside a Javascript

I am using scrapy to screen scrape data from a website. However, the data I wanted wasn't inside the html itself, instead, it is from a javascript. So, my question is: How to get the values (text ...
HeadAboutToExplode's user avatar
20 votes
3 answers
38k views

Scraping in Python - Preventing IP ban

I am using Python to scrape pages. Until now I didn't have any complicated issues. The site that I'm trying to scrape uses a lot of security checks and have some mechanism to prevent scraping. ...
RhymeGuy's user avatar
  • 2,112
15 votes
2 answers
13k views

Maintaining cookies between Mechanize requests

I'm trying to use the Ruby version of Mechanize to extract my employer's tickets from a ticket management system that we're moving away from that does not supply an API. Problem is, it seems ...
adamjford's user avatar
  • 7,538
14 votes
1 answer
29k views

Screen-scraping a windows application in c# [closed]

I need to scrape data from a windows application to run a query in another program. Does anyone know of a good starting point for me to do this in .NET?
Tony Boarman's user avatar
11 votes
4 answers
22k views

scrapy how to set referer url

I need to set the referer url, before scraping a site, the site uses refering url based Authentication, so it does not allow me to login if the referer is not valid. Could someone tell how to do this ...
vumaasha's user avatar
  • 2,795
10 votes
2 answers
13k views

Pass the user-agent through webdriver in Selenium

I am working on a website scraping project using Selenium in Python. When I open the homepage through a browser, it opens properly. But, when I try to open the webpage through webdriver() in Selenium,...
Kiran's user avatar
  • 8,250
9 votes
2 answers
35k views

How can I fix the following error AttributeError: 'dict' object has no attribute 'text' [duplicate]

I am a few months into programming. I am currently in the process of learning how to automate certain things in a project. My goal is to scrape text, src, and href and store the data in my site's ...
nothingness's user avatar
8 votes
4 answers
10k views

Python web scraping involving HTML tags with attributes

I'm trying to make a web scraper that will parse a web-page of publications and extract the authors. The skeletal structure of the web-page is the following: <html> <body> <div id="...
GobiasKoffi's user avatar
  • 4,064
8 votes
1 answer
3k views

I got error like Error in if (file.access(phantompath, 1) < 0) { : argument is of length zero when I use Phantomjs(port = 4567l)

my code is library(RSelenium) library(wdman) pDrv <- phantomjs(port = 4567L) below is what I got an error after runing above codes. checking phantomjs versions: BEGIN: PREDOWNLOAD ...
jeremyparty's user avatar
8 votes
4 answers
15k views

web scraping to fill out (and retrieve) search forms?

I was wondering if it is possible to "automate" the task of typing in entries to search forms and extracting matches from the results. For instance, I have a list of journal articles for which I would ...
hatmatrix's user avatar
  • 43.9k
8 votes
4 answers
11k views

Save all image files from a website

I'm creating a small app for myself where I run a Ruby script and save all of the images off of my blog. I can't figure out how to save the image files after I've identified them. Any help would be ...
Zack Shapiro's user avatar
  • 6,848
7 votes
3 answers
16k views

BeautifulSoup get_text does not strip all tags and JavaScript

I am trying to use BeautifulSoup to get text from web pages. Below is a script I've written to do so. It takes two arguments, first is the input HTML or XML file, the second output file. import sys ...
piokuc's user avatar
  • 26k
7 votes
10 answers
16k views

How can I screen scrape with Perl?

I need to display some values that are stored in a website, for that I need to scrape the website and fetch the content from the table. Any ideas?
6 votes
5 answers
4k views

PHP equivalent of PyQuery or Nokogiri? [closed]

Basically, I want to do some HTML screen scraping, but figuring out if it is possible in PHP. In Python, I would use PyQuery. In Ruby, I would use Nokogiri.
Swaroop C H's user avatar
6 votes
2 answers
10k views

Why does scrapy throw an error for me when trying to spider and parse a site?

The following code class SiteSpider(BaseSpider): name = "some_site.com" allowed_domains = ["some_site.com"] start_urls = [ "some_site.com/something/another/PRODUCT-...
bdd's user avatar
  • 3,436
6 votes
2 answers
5k views

Using Nokogiri to Split Content on BR tags

I have a snippet of code im trying to parse with nokogiri that looks like this: <td class="j"> <a title="title text1" href="http://link1.com">Link 1</a> (info1), Blah 1,<br&...
Mario Zigliotto's user avatar
6 votes
2 answers
3k views

Issue scraping page with "Load more" button with rvest

I want to obtain the links to the atms listed on this page: https://coinatmradar.com/city/345/bitcoin-atm-birmingham-uk/ Would I need to do something about the 'load more' button at the bottom of the ...
Jackb001's user avatar
5 votes
2 answers
7k views

Count number of results for a particular word on Twitter

To further a personal project of mine, I have been pondering how to count the number of results for a user specified word on Twitter. I have used their API extensively, but have not been able to come ...
Chris Ballance's user avatar
4 votes
3 answers
5k views

Convert a relative URL to an absolute URL with Simple HTML DOM?

When I'm scraping content from some pages, the script gives a relative URL. Is it possible to get a absolute URL with Simple HTML DOM?
Jean's user avatar
  • 41
4 votes
2 answers
9k views

Can a cURL based HTTP request imitate a browser based request completely?

This is a two part question. Q1: Can cURL based request 100% imitate a browser based request? Q2: If yes, what all options should be set. If not what extra does the browser do that cannot bee ...
gameover's user avatar
  • 11.9k
3 votes
1 answer
4k views

BeautifulSoup subpages of list with "load more" pagination

Quite new here, so apologies in advance. I'm looking to get a list of all company descriptions from https://angel.co/companies to play around with. The web-based parsing tools I've tried aren't ...
taylorhamcheese's user avatar
3 votes
5 answers
6k views

Get instagram followers

I want to parse a website's followers count with BeautifulSoup. This is what I have so far: username_extract = 'lazada_my' url = 'https://www.instagram.com/'+ username_extract r = requests.get(url) ...
calicationoflife's user avatar
2 votes
1 answer
8k views

Close a scrapy spider when a condition is met and return the output object

I have made a spider to get reviews from a page like this here using scrapy. I want product reviews only till a certain date(2nd July 2016 in this case). I want to close my spider as soon as the ...
Shubham Pandey's user avatar
2 votes
4 answers
2k views

What prevents me from using $.ajax to load another domain's html?

My domain: <!DOCTYPE html> <html> <head> <title>scrape</title> <script src="http://code.jquery.com/jquery-1.7.1.min.js"></script> </head> <body>...
jerome's user avatar
  • 4,929
2 votes
2 answers
6k views

How to scrape charts from a website with python?

EDIT: So I have save the script codes below to a text file but using re to extract the data still doesn't return me anything. My code is: file_object = open('source_test_script.txt', mode="r") ...
Ilumtics's user avatar
2 votes
4 answers
5k views

Screen Scraping of Image Links in PHP

I have a website that contains many different pages of products and each page has a certain amount of images in the same format across all pages. I want to be able to screen scrap each page's url so I ...
randomalbumtitle's user avatar
2 votes
4 answers
5k views

Screen scraping web page after delay

I'm trying to scrape a web page using C#, however after the page loads, it executes some JavaScript which loads more elements into the DOM which I need to scrape. A standard scraper simply grabs the ...
Justin's user avatar
  • 18k
0 votes
1 answer
353 views

Why is this HtmlAgilityPack operation invalid when there are, indeed, matching elements?

I get "InvalidOperationException > Message=Sequence contains no matching element" with the following code: private void buttonLoadHTML_Click(object sender, EventArgs e) { ...
B. Clay Shannon-B. Crow Raven's user avatar
0 votes
3 answers
2k views

Parse a .Net Page with Postbacks

I need to read data from an online database that's displayed using an aspx page from the UN. I've done HTML parsing before, but it was always by manipulating query-string values. In this case, the ...
user avatar

1
2
3 4 5
10