38

I need to remove all tags from a html with a bash script using the sed command. I tried with this

sed -r 's/[\<][\/]?[a-zA-Z0-9\=\"\-\#\.\& ]+[\/]?[\>]//g' $1

and whith this

sed -r 's/[\<][\/]?[.]*[\/]?[\\]?[\>]//g' $1

but I still miss something, any suggestions??

3
  • 2
    Here's my favorite answer concerning HTML and Regex ;-) stackoverflow.com/questions/1732348/… Nov 9, 2013 at 16:13
  • 1
    I perfectly know it, but this is a homework :-(
    – michste93
    Nov 9, 2013 at 16:15
  • 1
    My school time was long ago, but I think homework isn't necessarily to show a perfect answer. It's also to show some effort at all and to have the courage to say, that's all I've come up with. :-) Nov 9, 2013 at 16:26

3 Answers 3

103

You can either use one of the many HTML to text converters, use Perl regex if possible <.+?> or if it must be sed use <[^>]*>

sed -e 's/<[^>]*>//g' file.html

If there's no room for errors, use an HTML parser instead. E.g. when an element is spread over two lines

<div
>Lorem ipsum</div>

this regular expression will not work.


This regular expression consists of three parts <, [^>]*, >

  • search for opening <
  • followed by zero or more characters *, which are not the closing >
    [...] is a character class, when it starts with ^ look for characters not in the class
  • and finally look for closing >

The simpler regular expression <.*> will not work, because it searches for the longest possible match, i.e. the last closing > in an input line. E.g., when you have more than one tag in an input line

<name>Olaf</name> answers questions.

will result in

answers questions.

instead of

Olaf answers questions.

See also Repetition with Star and Plus, especially section Watch Out for The Greediness! and following, for a detailed explanation.

4
  • Works like a charm.. I suck at sed though, so could you please explain the <[^>]*> bit?
    – airstrike
    Mar 5, 2015 at 23:49
  • The [^>] is a character class which matches any character except > and the * says to repeat the match zero or more times. Otherwise, <.*> would match from the first < on the line to the last >, often spanning multiple tags.
    – tripleee
    Jun 4, 2016 at 9:49
  • Perfect, I needed to use grep -Ev "<.*>" to filter out HTML tags and the content in them. Thank you. Dec 3, 2020 at 22:50
  • Note: This answer doesn't consider script tags content. Apr 3 at 17:51
0

I know the OP asked for sed specifically but this page shows as a top result in Google even for non-sed searchers.

Perl one liner

cat - | perl -pe 's{\n}{ }g' | perl -pe 's{>}{>\n}g' | perl -pe 's{<}{\n<}g' | grep -v '<' | grep -v '^\s*$'

Feel free to edit this (I've marked it as community wiki), it's not perfect.

Explanation

Too much for me to type for now, but explainshell.com is a start.

Other notes

I'm surprised there isn't a mature tool out there that does this, just lots of messy npm command line tools. I'm not a fan of the amount of junk npm leaves behind. A golang precompiled single binary or something via brew install would be the ultimate dream

0

I've often used lynx -dump -nolist <URL> for the OP's purpose. However, you still get formatting, so you might want to additionally strip leading blanks on each line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.