Questions tagged [jsoup]

Jsoup is a Java HTML parser for extracting and manipulating HTML data, using the best of DOM, CSS, and jQuery-like methods.

Filter by
Sorted by
Tagged with
114 votes
15 answers
74k views

How do I preserve line breaks when using jsoup to convert html to plain text?

I have the following code: public class NewClass { public String noTags(String str){ return Jsoup.parse(str).text(); } public static void main(String args[]) { ...
Billy's user avatar
  • 1,141
104 votes
6 answers
68k views

Jsoup SocketTimeoutException: Read timed out

I get a SocketTimeoutException when I try to parse a lot of HTML documents using Jsoup. For example, I got a list of links : <a href="www.domain.com/url1.html">link1</a> <a ...
C. Maillard's user avatar
  • 1,041
57 votes
10 answers
113k views

How to "scan" a website (or page) for info, and bring it into my program?

Well, I'm pretty much trying to figure out how to pull information from a webpage, and bring it into my program (in Java). For example, if I know the exact page I want info from, for the sake of ...
James's user avatar
  • 5,632
52 votes
6 answers
64k views

jsoup posting and cookie

I'm trying to use jsoup to login to a site and then scrape information, I am running into in a problem, I can login successfully and create a Document from index.php but I cannot get other pages on ...
Gwindow's user avatar
  • 645
51 votes
3 answers
49k views

jsoup - strip all formatting and link tags, keep text only

Let's say i have a html fragment like this: <p> <span> foo </span> <em> bar <a> foobar </a> baz </em> </p> What i want to extract from that is: foo ...
WonderCsabo's user avatar
47 votes
4 answers
63k views

JSoup UserAgent, how to set it right?

I'm trying to parse the frontpage of facebook with JSoup but I always get the HTML Code for mobile devices and not the version for normal browsers(In my case Firefox 5.0). I'm setting my User Agent ...
Markus's user avatar
  • 531
46 votes
7 answers
47k views

How to add proxy support to Jsoup?

I am a beginner to Java and my first task is to parse some 10,000 URLs and extract some info out of it, for this I am using Jsoup and it's working fine. But now I want to add proxy support to it. The ...
Himanshu's user avatar
  • 1,443
46 votes
8 answers
67k views

Page content is loaded with JavaScript and Jsoup doesn't see it

One block on the page is filled with content by JavaScript and after loading page with Jsoup there is none of that inforamtion. Is there a way to get also JavaScript generated content when parsing ...
Eugene's user avatar
  • 4,352
43 votes
2 answers
29k views

Connection error: "org.jsoup.UnsupportedMimeTypeException: Unhandled content type"

When I try to open a link to parse with jsoup I get an error. Connection command: Document doc = Jsoup.connect("http://www.rfi.ro/podcast/emisiune/174/feed.xml") .timeout(10 * 1000)....
user2340897's user avatar
39 votes
3 answers
82k views

How to parse HTML table using jsoup?

I am trying to parse HTML using jsoup. This is my first time working with jsoup and I read some tutorial on it as well. Below is my HTML table which I am trying to parse - If you see my below table, ...
john's user avatar
  • 11.4k
39 votes
1 answer
4k views

How to parse data in Talend with Java (coming from a previously produced .txt file)?

I have a process in Talend which gets the search result of a page, saves the html and writes it into files, as seen here: Initially I had a two step process with parsing out the date from the HTML ...
ZedBrannigan's user avatar
38 votes
4 answers
37k views

Jsoup: how to get an image's absolute url?

Is there a way in jsoup to extract an image absolute url, much like one can get a link's absolute url? Consider the following image element found in http://www.example.com/ <img src="images/...
r0u1i's user avatar
  • 3,536
38 votes
1 answer
57k views

How to parse XML with jsoup

I am trying to parse XML with jsoup, but I can't find any examples on this task. My XML document looks like this: <?xml version="1.0" encoding="UTF-8"> <tests> <test> ...
JavaCake's user avatar
  • 4,085
36 votes
2 answers
34k views

(how) can I download an image using JSoup?

I already know where the image is, but for simplicity's sake I wanted to download the image using JSoup itself. (This is to simplify getting cookies, referrer, etc.) This is what I have so far: //...
user avatar
34 votes
3 answers
54k views

Jsoup select and iterate all elements

I will connect to a url through jsoup and get all the contents of it but the thing is if I select like, doc.select("body") its returning a single element but I want to get all the elements in the ...
Karthik 's user avatar
33 votes
2 answers
25k views

Jsoup select div having multiple classes

I am trying to select, using Jsoup, a <div> that has multiple classes: <div class="content-text right-align bold-font">...</div> The syntax for doing so, to the best of my ...
ef2011's user avatar
  • 10.5k
33 votes
4 answers
28k views

Does jsoup support xpath?

There's some work in progress related to adding xpath support to jsoup https://github.com/jhy/jsoup/pull/80. Is it working? How can I use it?
gguardin's user avatar
  • 551
31 votes
3 answers
38k views

Jsoup Cookies for HTTPS scraping

I am experimenting with this site to gather my username on the welcome page to learn Jsoup and Android. Using the following code Connection.Response res = Jsoup.connect("http://www.mikeportnoy.com/...
Brian's user avatar
  • 499
29 votes
10 answers
55k views

How to connect via HTTPS using Jsoup?

It's working fine over HTTP, but when I try and use an HTTPS source it throws the following exception: 10-12 13:22:11.169: WARN/System.err(332): javax.net.ssl.SSLHandshakeException: java.security....
jfisk's user avatar
  • 6,145
29 votes
7 answers
35k views

Jsoup.clean without adding html entities

I'm cleaning some text from unwanted HTML tags (such as <script>) by using String clean = Jsoup.clean(someInput, Whitelist.basicWithImages()); The problem is that it replaces for instance å ...
aioobe's user avatar
  • 415k
28 votes
4 answers
37k views

JSoup: Requesting JSON response

I'm using JSoup to authenticate then connect to a website. Some URL have a JSON response (because part of the site is in AJAX). Can JSoup handle JSON response ? Connection.Response doc = Jsoup....
Jean-Philippe Encausse's user avatar
28 votes
3 answers
31k views

How do I convert a document made in Jsoup (the Java html parser) into a string

I have a document that was made in jsoup that looks like this Document doc = Jsoup.connect("http://en.wikipedia.org/").get(); How do i convert that doc into a string.
Hudson Hughes's user avatar
28 votes
4 answers
14k views

Is it possible to convert HTML into XHTML with Jsoup 1.8.1?

String body = "<br>"; Document document = Jsoup.parseBodyFragment(body); document.outputSettings().escapeMode(EscapeMode.xhtml); String str = document.body().html(); System.out.println(str); ...
Henry's user avatar
  • 32.7k
27 votes
3 answers
50k views

How to POST Data into website using Jsoup

I am trying to POST data into website to make a login into the site using Jsoup , but its not working ? I am trying the code Document docs = Jsoup.connect("http://some.com/login") .data(...
Aspirant's user avatar
  • 2,238
27 votes
6 answers
31k views

How do I select this element in JSOUP?

This is the HTML structure: Element link = doc.select("div.subtabs p").first(); That does not seem to work. How do I select that p?
HackToHell's user avatar
  • 2,253
25 votes
4 answers
31k views

Java - Obtain text within script tag using Jsoup

I am using the Jsoup library to read a URL. This url has text within a few <script> tags. Is it possible for me to obtain the text within each <script> tag? Please note that I am not ...
M9A's user avatar
  • 3,188
25 votes
4 answers
10k views

How to avoid surrounding html head tags in Jsoup parse

Using Jsoup i try to parse the given html content. After Jsoup.parse() the html output append html, head and body tag to the input. I just want to ignore these. Sample Input: <p><b>This &...
Roshan's user avatar
  • 2,019
25 votes
3 answers
57k views

JSOUP select <div> with specific ID

I'm making a small Android application for a class where I find cancer-related events from the American Cancer Society's website. I've been using JSoup to get basic information about the events, and ...
Tom's user avatar
  • 911
24 votes
5 answers
48k views

Get element by class in JSoup

I try to get all info contained in div class named : bg_block_info, but instead i get info for another div class <div class="bg_block_info pad_20"> Why i'm getting it wrong ? Document doc = ...
develoops's user avatar
  • 545
24 votes
2 answers
44k views

how to parse a table from HTML using jsoup

<td width="10"></td> <td width="65"><img src="/images/sparks/NIFTY.png" /></td> <td width="65">5,390.85</td> <td width="65">5,428.15</td> <td ...
CyprUS's user avatar
  • 4,179
23 votes
4 answers
50k views

JSoup character encoding issue

I am using JSoup to parse content from http://www.latijnengrieks.com/vertaling.php?id=5368 . this is a third party website and does not specify proper encoding. i am using the following code to load ...
Hihaatje's user avatar
  • 263
23 votes
2 answers
29k views

Jsoup connection with basic access authentication

Is there a way in Jsoup to load a document from a website with basic access authentication?
user982940's user avatar
22 votes
9 answers
22k views

How can I extract only the main textual content from an HTML page?

Update Boilerpipe appears to work really well, but I realized that I don't need only the main content because many pages don't have an article, but only links with some short description to the ...
Renato Dinhani's user avatar
22 votes
2 answers
26k views

How to save a Jsoup Document to an HTML file?

I have used this method to retrieve a webpage into an org.jsoup.nodes.Document object: myDoc = Jsoup.connect(myURL).ignoreContentType(true).get(); How should I write this object to a HTML file? The ...
Ali Khezeli's user avatar
21 votes
3 answers
8k views

using jsoup with proguard closing force close

EDIT : MY PROGUARD VERSION IS 4.7 Today I tried include jsoup (version 1.7.1) in my android application, but it is causing me a lot of troubles. When I exported the signed apk with proguard turned on ...
android_newbie's user avatar
21 votes
3 answers
44k views

How to post form login using jsoup?

i want to login in here source code <HTML><HEAD><TITLE>:: Dhaka Electric Supply Company Limited (DESCO)::</TITLE> <META http-equiv=Content-Type content="text/html; ...
MD TAHMID HOSSAIN's user avatar
21 votes
3 answers
33k views

Java - Quickest way to check if URL exists

Hi I am writing a program that goes through many different URLs and just checks if they exist or not. I am basically checking if the error code returned is 404 or not. However as I am checking over ...
M9A's user avatar
  • 3,188
21 votes
1 answer
4k views

Jsoup exclude children from .text()

I have a problem similar to those: jQuery: exclude children from .text() Is it possible to achive it in JSoup?
emesx's user avatar
  • 12.6k
21 votes
1 answer
21k views

How to extract separate text nodes with Jsoup?

I have an element like this : <td> TextA <br/> TextB </td> How can I extract TextA and TextB separately?
M.M's user avatar
  • 1,353
21 votes
3 answers
9k views

Replace string with jsoup only in text portions

I have found several topics with similar questions and valuable answers, but I am still struggling with this: I want to parse some html with Jsoup so I can replace, for example, "changeme" with &...
Marcos Fernandez's user avatar
20 votes
5 answers
14k views

How to search for comments ("<!-- -->") using Jsoup?

I would like to remove those tags with their content from source HTML.
87element's user avatar
  • 1,919
20 votes
7 answers
36k views

Convert xPath to JSoup query

Does anyone know of an xPath to JSoup convertor? I get the following xPath from Chrome: //*[@id="docs"]/div[1]/h4/a and would like to change it into a Jsoup query. The path contains an href I'm ...
Josh's user avatar
  • 2,685
20 votes
2 answers
29k views

Jsoup Java HTML parser : Executing Javascript events

Can I fill out forms, execute events and Javascript functions in Jsoup? If yes how can I? Or should I go for another parser.
user avatar
20 votes
2 answers
19k views

How to extract absolute URL from relative HTML links using Jsoup?

I am using Jsoup to extract URL of an webpage. The href attribute of those URL's are relative like: <a href="/text">example</a> Here is my attempt: Document document = Jsoup.connect(url)...
sundhar's user avatar
  • 835
20 votes
1 answer
2k views

Selenium PhantomJS Java - Refused to evaluate a string because 'unsafe-eval' is not an allowed

I'm playing with Selenium and PhantomJS. I'm trying to draw all the elements from a web page. When I retrieve some web page and then I try to get the position of any web element I get this error when ...
Andrés Córdova's user avatar
19 votes
1 answer
9k views

JSoup.connect throws 403 error while apache.httpclient is able to fetch the content

I am trying to parse HTML dump of any given page. I used HTML Parser and also tried JSoup for parsing. I found useful functions in Jsoup but I am getting 403 error while calling Document doc = Jsoup....
instanceOfObject's user avatar
19 votes
6 answers
26k views

Jsoup like html parser for C++ [closed]

I have been writing some codes to get some data from some pages in Java and Jsoup was on of the best libraries to work with. But, Unfortunately I have to port the whole code to C/C++. But I a cannot ...
Writwick's user avatar
  • 2,143
19 votes
6 answers
30k views

Reading JSON Content

I'm using jsoup to scrape some HTML data and it's working out great. Now I need to pull some JSON content (only JSON, not HTML). Can I do this easily with jsoup or do I have to do it using another ...
Chromag's user avatar
  • 193
19 votes
4 answers
15k views

Parsing the html meta tag with jsoup library

Just started exploring the Jsoup library as i will use it for one of my projects. I tried googling but i could not find the exact answer that can help me. Here is my problem, i have an html file with ...
ivange's user avatar
  • 434
18 votes
3 answers
14k views

how to resolve jsoup error: unable to find valid certification path to requested target

I am trying to parse the html of the following URL: https://www.smuc.ac.kr/mbs/smuc/jsp/board/list.jsp?boardId=6993&id=smuc_040100000000 I'm getting the following error: sun.security.validator....
Behzod Muslimov's user avatar

1
2 3 4 5
136