9

There are lots of examples of how to strip HTML tags from a document using Ruby, Hpricot and Nokogiri have inner_text methods that remove all HTML for you easily and quickly.

What I am trying to do is the opposite, remove all the text from an HTML document, leaving just the tags and their attributes.

I considered looping through the document setting inner_html to nil but then really you'd have to do this in reverse as the first element (root) has an inner_html of the entire rest of the document, so ideally I'd have to start at the inner most element and set inner_html to nil whilst moving up through the ancestors.

Does anyone know a neat little trick for doing this efficiently? I was thinking perhaps regex's might do it but probably not as efficiently as an HTML tokenizer/parser might.

2
  • Are you going to have to deal with bad markup? (unescaped entities, etc.)
    – Neall
    Sep 30, 2009 at 11:57
  • It's possible - the markup I'm processing comes from end users so can't be relied on. Sep 30, 2009 at 13:09

4 Answers 4

38

This works too:

doc = Nokogiri::HTML(your_html)
doc.xpath("//text()").remove
1
  • 1
    Excellent! Toss a .to_s on the end to get a string and you're good to go!
    – Sam Soffes
    Mar 5, 2012 at 0:28
3

You can scan the string to create an array of "tokens", and then only select those that are html tags:

>> some_html
=> "<div>foo bar</div><p>I like <em>this</em> stuff <a href='http://foo.bar'> long time</a></p>"
>> some_html.scan(/<\/?[^>]+>|[\w\|`~!@#\$%^&*\(\)\-_\+=\[\]{}:;'",\.\/?]+|\s+/).select { |t| t =~ /<\/?[^>]+>/ }.join("")
=> "<div></div><p><em></em><a href='http://foo.bar'></a></p>"

==Edit==

Or even better, just scan for html tags ;)

>> some_html.scan(/<\/?[^>]+>/).join("")
=> "<div></div><p><em></em><a href='http://foo.bar'></a></p>"
3

To grab everything not in a tag, you can use nokogiri like this:

doc.search('//text()').text

Of course, that will grab stuff like the contents of <script> or <style> tags, so you could also remove blacklisted tags:

blacklist = ['title', 'script', 'style']
nodelist = doc.search('//text()')
blacklist.each do |tag|
  nodelist -= doc.search('//' + tag + '/text()')
end
nodelist.text

You could also whitelist if you preferred, but that's probably going to be more time-intensive:

whitelist = ['p', 'span', 'strong', 'i', 'b']  #The list goes on and on...
nodelist = Nokogiri::XML::NodeSet.new(doc)
whitelist.each do |tag|
  nodelist += doc.search('//' + tag + '/text()')
end
nodelist.text

You could also just build a huge XPath expression and do one search. I honestly don't know which way is faster, or if there is even an appreciable difference.

0
0

I just came up with this, but @andre-r's solution is soo much better!

#!/usr/bin/env ruby

require 'nokogiri'

def strip_text doc
  Nokogiri(doc).tap { |doc|
    doc.traverse do |node|
      node.content = nil if node.text?
    end
  }.to_s
end

require 'test/unit'
require 'yaml'
class TestHTMLStripping < Test::Unit::TestCase
  def test_that_all_text_gets_strippped_from_the_document
    dirty, clean = YAML.load DATA
    assert_equal clean, strip_text(dirty)
  end
end
__END__
---
- |
  <!DOCTYPE html>
  <html xmlns='http://www.w3.org/1999/xhtml' xml:lang='en' lang='en'>
    <head>
        <meta http-equiv='Content-type'     content='text/html; charset=UTF-8' />
        <title>Test HTML Document</title>
        <meta http-equiv='content-language' content='en' />
    </head>
    <body>
        <h1>Test <abbr title='Hypertext Markup Language'>HTML</abbr> Document</h1>
        <div class='main'>
            <p>
                <strong>Test</strong> <abbr title='Hypertext Markup Language'>HTML</abbr> <em>Document</em>
            </p>
        </div>
    </body>
  </html>
- |
  <!DOCTYPE html>
  <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  <title></title>
  <meta http-equiv="content-language" content="en">
  </head>
  <body><h1><abbr title="Hypertext Markup Language"></abbr></h1><div class="main"><p><strong></strong><abbr title="Hypertext Markup Language"></abbr><em></em></p></div></body>
  </html>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.