Questions tagged [webharvest]

Web-Harvest is Open Source Web Data Extraction tool written in Java.

webharvest
Filter by
Sorted by
Tagged with
0 votes
2 answers
499 views

character (0) after scraping webpage in read_html

I'm trying to scrape "1,335,000" from the screenshot below (the number is at the bottom of the screenshot). I wrote the following code in R. t2<-read_html("https://fortune.com/...
Xian Zhao's user avatar
-2 votes
2 answers
88 views

How can I add the next list to do a sum and avarage?

I got the following code: def arrayOfInts = [actual_speed_mobile_1.toInteger(), actual_speed_mobile_2.toInteger(), actual_speed_mobile_3.toInteger(), actual_speed_mobile_4.toInteger(), ...
iollo's user avatar
  • 47
0 votes
1 answer
95 views

Gson.toJson() method returns string embedded with "data"

I am working on Workfusion which is one of the automation tool and it consists of selenium, java and Web -harvest technology. Below is the snipped of my code. <while condition="${continue_flg}" ...
Nitin Rathod's user avatar
2 votes
1 answer
11k views

how to fix groovy.lang.MissingMethodException: No signature of method: java.util.ArrayList.get() is applicable for argument types: () values: []

I trying to use this method in groovy, groupedDocs = reader.selectGroupedDocs(last_update_date.toString()).get(); And this is my java code part for "selectGroupedDocs" method, private List<Map&...
Sachika Sandeepa's user avatar
0 votes
1 answer
226 views

Rvest returns zero list

I want to download all links/ titles of papers from the web using rvest. I used the following script but it is not the list is zero. Any suggestions? library(rvest) 1. Download the HTML and turn it ...
Ehsan 's user avatar
  • 11
1 vote
0 answers
419 views

Web scraping with rvest - login not working - flightradar24.com

I'm trying to harvest data from www.flightradar24.com using rvest. I've got a subscription, so I want to log in and get access to more data. This is the code I'm using to log in (I'm using my email ...
Eloy's user avatar
  • 19
0 votes
2 answers
2k views

Angular 4 how to request a web page content as a json object

I am trying to request a web page with a http call and harwest the data. I could avoid the cross-origin with a chrome plug-in but still when I make the request, the response is always "null". How ...
tlq's user avatar
  • 919
1 vote
0 answers
210 views

http webharvest tag not working with http-param parameters

I am trying the below code in webharvest <var-def name="url">https://www.athome.com/on/demandware.store/Sites-athome-Site/default/Stores-FindByZip?</var-def> <var-def name="urlResult"...
Here s Toimprove's user avatar
1 vote
0 answers
100 views

Invoke webharvest function from JavaScript function

I have created a webharvest function. I am able to invoke the function using webharvest code. My challenge is, need to invoke that webharvest function from a JavaScript function. Is it possible? For ...
Sunil Prabakar's user avatar
1 vote
1 answer
109 views

Getting data from Wep Pages using Jsoup Java

Its my first qustion in this site and hope to stay longer :=) I have read a lot of article and examine many kind of example about taking specific datas from web site using Jsoup. Alread, I could ...
samio's user avatar
  • 11
1 vote
1 answer
691 views

Getting response headers with Java, encoding issue

I am using Webharvest to download a file from a website and take its original name. The Java code that I am working with is: import org.apache.commons.httpclient.Header; import org....
linderman's user avatar
  • 158
0 votes
0 answers
140 views

XSLT error when getting variable [duplicate]

I am trying to assign variables into xsl file and use them later in this file. Here is how i assign them: <xsl:choose> <xsl:when test="matches(normalize-space(tokenize((//span[@class='...
linderman's user avatar
  • 158
1 vote
1 answer
476 views

XPath for text after a div?

How could I extract the number "-105" with XPath 1.0/2.0? <td class="alignRight bookColumn" rel="page1" style=""> <div id="_Div_Line_5_523714_901_3_119">-115</div> <div id="...
Fernando's user avatar
1 vote
1 answer
139 views

How to webscrape share counts in R

I am trying to download the share count from the left SumoMe plugin of this website http://www.r-bloggers.com/erum-2016-first-european-conference-for-the-programming-language-r/ I try to use R code ...
Marcin's user avatar
  • 7,934
1 vote
1 answer
161 views

Webharvest crawler script not creating XML file

I'm hoping someone can point out my (probably stupid) problem with this script. I'm trying to crawl a website to get the posts on the site and to load this into an XML document. I have tried to ...
zag2010's user avatar
  • 469
0 votes
1 answer
122 views

How to replace string after extraction from WebHarvest?

I wanted to insert the records I had extracted from website to DB, but the extraction text contained the symbol apostrophe, and had caused me syntax error during sql insertion. May I know how to ...
Jazz's user avatar
  • 21
2 votes
1 answer
122 views

Trying to grab information in Child Link using WebHarvest

I would like to grab the information of each child link, but the program shows error. Below are my full config file. The error is Caused by: org.xml.sax.SAXParseException; lineNumber: 6; columnNumber: ...
Jazz's user avatar
  • 21
0 votes
1 answer
445 views

How to get large pictures in Google image

I want to collect pictures from Google image search. However, I am constantly notified with an error. For example, the URL https://www.google.com/search?q=banana&amp;hl=en&gws_rd=ssl&tbm=...
user3672968's user avatar
0 votes
1 answer
59 views

What is the difference between the exitExecution() and stopExecution() in Webharvest Scraper class

I want to know what is the difference between the scraper.exitExecution() and scraper.stopExecution() and scraper.finishExecutingProcessor() I have tried looking in to the java doc, I could ...
codeMan's user avatar
  • 5,748
0 votes
1 answer
319 views

WebHarvest need 50 result in one request

I am new with this language and I am stuck in a simple task. Basically I would like to get 50 results instead of the basic 10 results that the searcher give me as a basic result. This will be the ...
Kein-Wai's user avatar
1 vote
2 answers
552 views

Disabling XML validation in WebHarvest

I have a mobile application already published in the Apple's app store. This SPI client app uses a Rest API in the server side to retrieve real time information regarding buses arrivals in a specific ...
Julio Helden's user avatar
0 votes
1 answer
276 views

Adding numbers to variables in WebHarvest

I should start off by saying that I'm very new to javascript. I need to feed a bunch of urls to webharvest based on a number. It's a long story why, but my url structure looks kind of like this: http:...
Jangari's user avatar
  • 848
1 vote
1 answer
605 views

Webharvest if/else and try/catch always succeeding

I'm working on a project where I need to harvest some data from website, so I'm using webharvest. I'm running into a problem where the data I'm harvesting (comments from news websites) is sometimes ...
Jangari's user avatar
  • 848
0 votes
1 answer
208 views

How to output the browser User-Agent String using Web-Harvest library in Java

Testing some issues with an application that has been created using the Web-Harvest library to no avail. Our network environment has a mix of proxies and other filters and I am trying to rule out a ...
FailHorse's user avatar
1 vote
1 answer
353 views

Scrape data from website using webharvest

I am trying to scrape all html pages from website "http://www.tecomdirectory.com/" using webharvest. But the script fails to grab all the html pages and only scrapes few of the html pages. I ...
user3717330's user avatar
0 votes
1 answer
218 views

extract data from Pdf using Web harvesting

How can i extract data from PDF using Web Harvesting? I am getting all the relevant PDFs url in a page but i am not been able to extract data out of those Pdf.I am using Web Harvest version 2.0 for ...
user3536614's user avatar
1 vote
1 answer
286 views

WebHarvest - Scrape data using authentication

I am using the WebHarvest tool to scrape web data from a few websites. I have gone through the examples, but was not able to find a way to authenticate in websites and then scrape data from them. Can ...
ArinCool's user avatar
  • 1,730
0 votes
2 answers
415 views

collect data from child links by web harvest

Is there any way to collect data from child link for Web Harvest? Below is a xml segment I use: <loop item="item" index="i"> <list><var name="products"/></list> ...
John Shen's user avatar
0 votes
1 answer
7k views

xpst0003 xquery syntax error

I want to query a webpage through following x query code. please help me. And it gives me following errors: XPST0003: XQuery syntax error in #...//json//sentences//trans); let#: expected "return",...
codeht_1's user avatar
-1 votes
1 answer
503 views

HtmlUnit scraping google+ page javascript. Click show more button not working

i am trying to scrap this page https://plus.google.com/115016587855962294424/about. Everything works fine but when i try to click show more to load more reviews nothing happens here is my code ...
tariq.freeman's user avatar
1 vote
2 answers
454 views

is it possible to use a string variable as a tag in xquery

Can I use something like following in xquery? Basically I want to use string variable as a tag. I am using web harvest to extract data. Any help or pointers appreciated. declare variable $doc as node(...
trailblazer's user avatar
  • 1,421
0 votes
1 answer
222 views

variable as a tag for xml output

I am using a tool web harvest to scrape data from a website. I used xquery to output data as an xml file. I need to use a variable tag, for which I tried using the following: for $i in (4 to ($count -...
trailblazer's user avatar
  • 1,421
0 votes
1 answer
726 views

Crawling a javascript variable inside HTML file with web-harvest

I have a question (dummy maybe). Suppose that I have the following snippet of code inside an html page with a bunch of other things (other script tags and html tags): <script type="text/javascript"...
Julio Helden's user avatar
2 votes
3 answers
2k views

Web scraping in PHP - working with some URLs but fails with others

I am doing web scraping with curl for a linkedin profile page. If we try to extract data from this(http://in.linkedin.com/in/ratneshdwivedi) URL which is public, it's working. When I am logged in to ...
ratnesh dwivedi's user avatar
0 votes
1 answer
201 views

how to not select somedata using xpath in webharvest

I am using webharvest with xquery to get a data from a website. I have the 2 xquery variables with the following data $text: <p> <strong>Psoria-Shield Inc.</strong> (<a href="...
codeMan's user avatar
  • 5,748
0 votes
1 answer
479 views

How to subtract a substring from a string in web harvest

I am new to webharvest and am using it to get the article data from a website, using the following statement: let $text := data($doc//div[@id="articleBody"]) and this is the data that I get from ...
codeMan's user avatar
  • 5,748
0 votes
1 answer
121 views

How to strip a part of the text obtained from web harvest

I am new to webharvest and am using it to get the article data from a website, using the following statement: let $text := data($doc//div[@id="articleBody"]) and this is the data that I get from ...
codeMan's user avatar
  • 5,748
0 votes
1 answer
1k views

Web-Harvest: grabbing multiple url's from a list

What I'm trying is to get multiple webpages from a predefined list. Here is the code: <?xml version="1.0" encoding="UTF-8"?> <config> <script> <![CDATA[ ...
Msam85's user avatar
  • 4,104
0 votes
2 answers
1k views

IO error during HTTP execution

I used the web-harvest for 5 month and I tried to get the content of webs with this syntax: <var-def name="raw"> <html-to-xml outputtype="pretty" usecdata="false"> <http url="${URL....
user329765's user avatar
1 vote
2 answers
1k views

Trying to Extract URL's from a Website Using Web Harvest

I'm trying to extract the URL's of a website that doesn't have a sitemap. I'm using the Web Harvest tool I have no idea about Java or coding. Could someone please help me out with using this tool. ...
Caleb's user avatar
  • 11
0 votes
1 answer
785 views

how to save web-harvest data to database

I am scrapping the data using web-harvest tool and i am getting the required data i.e. name and price of the product. here is my config file. <include path="functions.xml"/> <!-- collects ...
sahithi's user avatar
  • 25
1 vote
2 answers
1k views

Webharvest If and null test

I'm trying to make my program check the return of an xpath expression and if it is null it should try a different one, how do I do this? I have tried all the examples on the website and the blank ...
user2330657's user avatar
-1 votes
1 answer
1k views

Limiting list returned by xpath [duplicate]

I am trying to use xpath in WebHarvest and I am able to receive a large list of data, however I only need the first 5 strings returned. <var-def name="googleProducts"> <html-to-xml> ...
user2330657's user avatar
0 votes
1 answer
618 views

WebHarvest XML not well formed

I am using WebHarvest to try to receive data from Woot.com and I'm getting a few different errors. I am able to get the website with the first process, but when I try to test xpath inside of the ...
user2330657's user avatar
2 votes
2 answers
1k views

web harvest - scraping an url

I am using web harvest. However, I want to scrape data from the URL: http://derstandard.at/anzeiger/immoweb/Suchergebnis.aspx?Regionen=9&Bezirke=&Arten=&AngebotTyp=&timestamp=...
user2051347's user avatar
  • 1,649
1 vote
0 answers
115 views

How to run Webharvest on Google App engine?

I have developed a scraper with web harvest(http://web-harvest.sourceforge.net/). However, I want to let the program run on a server. Is it possible to run this system on google app engine? Thx for ...
user2051347's user avatar
  • 1,649
0 votes
2 answers
1k views

Using regex in web harvest xml

I'm using web harvest to scrap some e-commerce site.I'm iterating over the search page and getting each product details in output xml.But now I want to use regular expression in anchor(a) tag while ...
Ashwini's user avatar
  • 41
1 vote
1 answer
630 views

WebHarvest can't find response headers

I'm working with WebHarvest to fetch data from a site that requires logging in. It's setup like this: Page 1 = Login page Page 2 = Login validation page Page 3 = Statistics page On page 2 a ...
SiggeLund's user avatar
0 votes
1 answer
695 views

Web harvest -- remove unusual characters

I'm trying to scrape a page that has some spaces after the anchors: </a>&nbsp;&nbsp;|&nbsp;&nbsp; I can't seem to find a way to specify the text, and I either trigger a ...
user991945's user avatar
2 votes
2 answers
452 views

webharvest not retrieving data

I have webharvest running without errors, but when I open the XML file it does not have the right data, it just prints it out. here is my code: <?xml version="1.0" encoding="UTF-8"?> <...
stacktraceyo's user avatar
  • 1,235