Questions tagged [jsoup]
Jsoup is a Java HTML parser for extracting and manipulating HTML data, using the best of DOM, CSS, and jQuery-like methods.
6,795
questions
114
votes
15
answers
74k
views
How do I preserve line breaks when using jsoup to convert html to plain text?
I have the following code:
public class NewClass {
public String noTags(String str){
return Jsoup.parse(str).text();
}
public static void main(String args[]) {
...
104
votes
6
answers
68k
views
Jsoup SocketTimeoutException: Read timed out
I get a SocketTimeoutException when I try to parse a lot of HTML documents using Jsoup.
For example, I got a list of links :
<a href="www.domain.com/url1.html">link1</a>
<a ...
57
votes
10
answers
113k
views
How to "scan" a website (or page) for info, and bring it into my program?
Well, I'm pretty much trying to figure out how to pull information from a webpage, and bring it into my program (in Java).
For example, if I know the exact page I want info from, for the sake of ...
52
votes
6
answers
64k
views
jsoup posting and cookie
I'm trying to use jsoup to login to a site and then scrape information, I am running into in a problem, I can login successfully and create a Document from index.php but I cannot get other pages on ...
51
votes
3
answers
49k
views
jsoup - strip all formatting and link tags, keep text only
Let's say i have a html fragment like this:
<p> <span> foo </span> <em> bar <a> foobar </a> baz </em> </p>
What i want to extract from that is:
foo ...
47
votes
4
answers
63k
views
JSoup UserAgent, how to set it right?
I'm trying to parse the frontpage of facebook with JSoup but I always get the HTML Code for mobile devices and not the version for normal browsers(In my case Firefox 5.0).
I'm setting my User Agent ...
46
votes
7
answers
47k
views
How to add proxy support to Jsoup?
I am a beginner to Java and my first task is to parse some 10,000 URLs and extract some info out of it, for this I am using Jsoup and it's working fine.
But now I want to add proxy support to it. The ...
46
votes
8
answers
67k
views
Page content is loaded with JavaScript and Jsoup doesn't see it
One block on the page is filled with content by JavaScript and after loading page with Jsoup there is none of that inforamtion. Is there a way to get also JavaScript generated content when parsing ...
43
votes
2
answers
29k
views
Connection error: "org.jsoup.UnsupportedMimeTypeException: Unhandled content type"
When I try to open a link to parse with jsoup I get an error.
Connection command:
Document doc = Jsoup.connect("http://www.rfi.ro/podcast/emisiune/174/feed.xml")
.timeout(10 * 1000)....
39
votes
3
answers
82k
views
How to parse HTML table using jsoup?
I am trying to parse HTML using jsoup. This is my first time working with jsoup and I read some tutorial on it as well. Below is my HTML table which I am trying to parse -
If you see my below table, ...
39
votes
1
answer
4k
views
How to parse data in Talend with Java (coming from a previously produced .txt file)?
I have a process in Talend which gets the search result of a page, saves the html and writes it into files, as seen here:
Initially I had a two step process with parsing out the date from the HTML ...
38
votes
4
answers
37k
views
Jsoup: how to get an image's absolute url?
Is there a way in jsoup to extract an image absolute url, much like one can get a link's absolute url?
Consider the following image element found in http://www.example.com/
<img src="images/...
38
votes
1
answer
57k
views
How to parse XML with jsoup
I am trying to parse XML with jsoup, but I can't find any examples on this task.
My XML document looks like this:
<?xml version="1.0" encoding="UTF-8">
<tests>
<test>
...
36
votes
2
answers
34k
views
(how) can I download an image using JSoup?
I already know where the image is, but for simplicity's sake I wanted to download the image using JSoup itself. (This is to simplify getting cookies, referrer, etc.)
This is what I have so far:
//...
34
votes
3
answers
54k
views
Jsoup select and iterate all elements
I will connect to a url through jsoup and get all the contents of it but the thing is if I select like,
doc.select("body")
its returning a single element but I want to get all the elements in the ...
33
votes
2
answers
25k
views
Jsoup select div having multiple classes
I am trying to select, using Jsoup, a <div> that has multiple classes:
<div class="content-text right-align bold-font">...</div>
The syntax for doing so, to the best of my ...
33
votes
4
answers
28k
views
Does jsoup support xpath?
There's some work in progress related to adding xpath support to jsoup https://github.com/jhy/jsoup/pull/80.
Is it working?
How can I use it?
31
votes
3
answers
38k
views
Jsoup Cookies for HTTPS scraping
I am experimenting with this site to gather my username on the welcome page to learn Jsoup and Android. Using the following code
Connection.Response res = Jsoup.connect("http://www.mikeportnoy.com/...
29
votes
10
answers
55k
views
How to connect via HTTPS using Jsoup?
It's working fine over HTTP, but when I try and use an HTTPS source it throws the following exception:
10-12 13:22:11.169: WARN/System.err(332): javax.net.ssl.SSLHandshakeException: java.security....
29
votes
7
answers
35k
views
Jsoup.clean without adding html entities
I'm cleaning some text from unwanted HTML tags (such as <script>) by using
String clean = Jsoup.clean(someInput, Whitelist.basicWithImages());
The problem is that it replaces for instance å ...
28
votes
4
answers
37k
views
JSoup: Requesting JSON response
I'm using JSoup to authenticate then connect to a website. Some URL have a JSON response (because part of the site is in AJAX). Can JSoup handle JSON response ?
Connection.Response doc = Jsoup....
28
votes
3
answers
31k
views
How do I convert a document made in Jsoup (the Java html parser) into a string
I have a document that was made in jsoup that looks like this
Document doc = Jsoup.connect("http://en.wikipedia.org/").get();
How do i convert that doc into a string.
28
votes
4
answers
14k
views
Is it possible to convert HTML into XHTML with Jsoup 1.8.1?
String body = "<br>";
Document document = Jsoup.parseBodyFragment(body);
document.outputSettings().escapeMode(EscapeMode.xhtml);
String str = document.body().html();
System.out.println(str);
...
27
votes
3
answers
50k
views
How to POST Data into website using Jsoup
I am trying to POST data into website to make a login into the site using Jsoup , but its not working ?
I am trying the code
Document docs = Jsoup.connect("http://some.com/login")
.data(...
27
votes
6
answers
31k
views
How do I select this element in JSOUP?
This is the HTML structure:
Element link = doc.select("div.subtabs p").first();
That does not seem to work. How do I select that p?
25
votes
4
answers
31k
views
Java - Obtain text within script tag using Jsoup
I am using the Jsoup library to read a URL. This url has text within a few <script> tags. Is it possible for me to obtain the text within each <script> tag? Please note that I am not ...
25
votes
4
answers
10k
views
How to avoid surrounding html head tags in Jsoup parse
Using Jsoup i try to parse the given html content. After Jsoup.parse() the html output append html, head and body tag to the input. I just want to ignore these.
Sample Input:
<p><b>This &...
25
votes
3
answers
57k
views
JSOUP select <div> with specific ID
I'm making a small Android application for a class where I find cancer-related events from the American Cancer Society's website. I've been using JSoup to get basic information about the events, and ...
24
votes
5
answers
48k
views
Get element by class in JSoup
I try to get all info contained in div class named : bg_block_info, but instead i get info for another div class <div class="bg_block_info pad_20"> Why i'm getting it wrong ?
Document doc = ...
24
votes
2
answers
44k
views
how to parse a table from HTML using jsoup
<td width="10"></td>
<td width="65"><img src="/images/sparks/NIFTY.png" /></td>
<td width="65">5,390.85</td>
<td width="65">5,428.15</td>
<td ...
23
votes
4
answers
50k
views
JSoup character encoding issue
I am using JSoup to parse content from http://www.latijnengrieks.com/vertaling.php?id=5368 . this is a third party website and does not specify proper encoding. i am using the following code to load ...
23
votes
2
answers
29k
views
Jsoup connection with basic access authentication
Is there a way in Jsoup to load a document from a website with basic access authentication?
22
votes
9
answers
22k
views
How can I extract only the main textual content from an HTML page?
Update
Boilerpipe appears to work really well, but I realized that I don't need only the main content because many pages don't have an article, but only links with some short description to the ...
22
votes
2
answers
26k
views
How to save a Jsoup Document to an HTML file?
I have used this method to retrieve a webpage into an org.jsoup.nodes.Document object:
myDoc = Jsoup.connect(myURL).ignoreContentType(true).get();
How should I write this object to a HTML file?
The ...
21
votes
3
answers
8k
views
using jsoup with proguard closing force close
EDIT : MY PROGUARD VERSION IS 4.7
Today I tried include jsoup (version 1.7.1) in my android application, but it is causing me a lot of troubles. When I exported the signed apk with proguard turned on ...
21
votes
3
answers
44k
views
How to post form login using jsoup?
i want to login in here
source code
<HTML><HEAD><TITLE>:: Dhaka Electric Supply Company Limited (DESCO)::</TITLE>
<META http-equiv=Content-Type content="text/html; ...
21
votes
3
answers
33k
views
Java - Quickest way to check if URL exists
Hi I am writing a program that goes through many different URLs and just checks if they exist or not. I am basically checking if the error code returned is 404 or not. However as I am checking over ...
21
votes
1
answer
4k
views
Jsoup exclude children from .text()
I have a problem similar to those:
jQuery: exclude children from .text()
Is it possible to achive it in JSoup?
21
votes
1
answer
21k
views
How to extract separate text nodes with Jsoup?
I have an element like this :
<td> TextA <br/> TextB </td>
How can I extract TextA and TextB separately?
21
votes
3
answers
9k
views
Replace string with jsoup only in text portions
I have found several topics with similar questions and valuable answers, but I am still struggling with this:
I want to parse some html with Jsoup so I can replace, for example,
"changeme"
with
&...
20
votes
5
answers
14k
views
How to search for comments ("<!-- -->") using Jsoup?
I would like to remove those tags with their content from source HTML.
20
votes
7
answers
36k
views
Convert xPath to JSoup query
Does anyone know of an xPath to JSoup convertor? I get the following xPath from Chrome:
//*[@id="docs"]/div[1]/h4/a
and would like to change it into a Jsoup query. The path contains an href I'm ...
20
votes
2
answers
29k
views
Jsoup Java HTML parser : Executing Javascript events
Can I fill out forms, execute events and Javascript functions in Jsoup? If yes how can I? Or should I go for another parser.
20
votes
2
answers
19k
views
How to extract absolute URL from relative HTML links using Jsoup?
I am using Jsoup to extract URL of an webpage. The href attribute of those URL's are relative like:
<a href="/text">example</a>
Here is my attempt:
Document document = Jsoup.connect(url)...
20
votes
1
answer
2k
views
Selenium PhantomJS Java - Refused to evaluate a string because 'unsafe-eval' is not an allowed
I'm playing with Selenium and PhantomJS. I'm trying to draw all the elements from a web page. When I retrieve some web page and then I try to get the position of any web element I get this error when ...
19
votes
1
answer
9k
views
JSoup.connect throws 403 error while apache.httpclient is able to fetch the content
I am trying to parse HTML dump of any given page. I used HTML Parser and also tried JSoup for parsing.
I found useful functions in Jsoup but I am getting 403 error while calling Document doc = Jsoup....
19
votes
6
answers
26k
views
Jsoup like html parser for C++ [closed]
I have been writing some codes to get some data from some pages in Java and Jsoup was on of the best libraries to work with. But, Unfortunately I have to port the whole code to C/C++. But I a cannot ...
19
votes
6
answers
30k
views
Reading JSON Content
I'm using jsoup to scrape some HTML data and it's working out great. Now I need to pull some JSON content (only JSON, not HTML). Can I do this easily with jsoup or do I have to do it using another ...
19
votes
4
answers
15k
views
Parsing the html meta tag with jsoup library
Just started exploring the Jsoup library as i will use it for one of my projects. I tried googling but i could not find the exact answer that can help me. Here is my problem, i have an html file with ...
18
votes
3
answers
14k
views
how to resolve jsoup error: unable to find valid certification path to requested target
I am trying to parse the html of the following URL:
https://www.smuc.ac.kr/mbs/smuc/jsp/board/list.jsp?boardId=6993&id=smuc_040100000000
I'm getting the following error:
sun.security.validator....