6

I want to save text I scrape from various sources without the HTML tags that are on it, but also keeping as much of the structure as I reasonably can.

Markdown seems to be the solution to this (or possibly MultiMarkdown).

There is a question which offers a suggestion on converting from HTML to markdown, but I want to specify some specific things:

  • ALL links (including images) are referenced at the END only (i.e. no inline urls)
  • NO embeded HTML (I'm not even 100% sure yet how I'd like to deal with difficult HTML... but it won't be embeded!)

So my question is as stated in the title: Is there a decent, customisable, HTML to Markdown Java API?

2 Answers 2

2

You could try adapting HtmlCleaner which provides a workable interface onto the DOM:

TagNode root = htmlCleaner.clean( stream );
Object[] found = root.evaluateXPath( "//div[id='something']" );
if( found.length > 0 && found instanceof TagNode ) {
    ((TagNode)found[0]).removeFromTree();
}

This would allow you to structure your output stream in any format that you want using a fairly simple API.

1

There is a great library for JS called Turndown, you can try it online here. It can be partially customized. For example, links can be referenced at the end. And as far as I know there is no embedded html, everything is transformed.

I needed it for Java (as the linked question), so I ported it. The library for Java is called CopyDown, it has the same test suite as Turndown.

To install with gradle:

dependencies {
        compile 'io.github.furstenheim:copy_down:1.0'
}

Then to use it:

CopyDown converter = new CopyDown();
String myHtml = "<h1>Some title</h1><div>Some html<p>Another paragraph</p></div>";
String markdown = converter.convert(myHtml);
System.out.println(markdown);
> Some title\n==========\n\nSome html\n\nAnother paragraph\n

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.