7

I need some help a couple of questions, using bash tools

  1. I want to remove empty xml tags from a file eg:
 <CreateOfficeCode>
      <OperatorId>ve</OperatorId>
      <OfficeCode>1234</OfficeCode>
      <CountryCodeLength>0</CountryCodeLength>
      <AreaCodeLength>3</AreaCodeLength>
      <Attributes></Attributes>
      <ChargeArea></ChargeArea>
 </CreateOfficeCode>

to become:

 <CreateOfficeCode>
      <OperatorId>ve</OperatorId>
      <OfficeCode>1234</OfficeCode>
      <CountryCodeLength>0</CountryCodeLength>
      <AreaCodeLength>3</AreaCodeLength>
 </CreateOfficeCode>

for this I have done so by this command

sed -i '/><\//d' file

which is not so strict, its more like a trick, something more appropriate would be to find the <pattern></pattern> and remove it. Suggestion?

  1. Second, how to go from:
 <CreateOfficeGroup>
       <CreateOfficeName>John</CreateOfficeName>
       <CreateOfficeCode>
       </CreateOfficeCode>
 </CreateOfficeGroup>

to:

 <CreateOfficeGroup>
       <CreateOfficeName>John</CreateOfficeName>
 </CreateOfficeGroup>
  1. As a whole thing? from:
 <CreateOfficeGroup>
       <CreateOfficeName>John</CreateOfficeName>
       <CreateOfficeCode>
            <OperatorId>ve</OperatorId>
            <OfficeCode>1234</OfficeCode>
            <CountryCodeLength>0</CountryCodeLength>
            <AreaCodeLength>3</AreaCodeLength>
            <Attributes></Attributes>
            <ChargeArea></ChargeArea>
       </CreateOfficeCode>
       <CreateOfficeSize>
            <Chairs></Chairs>
            <Tables></Tables>
       </CreateOfficeSize>
 </CreateOfficeGroup>

to:

 <CreateOfficeGroup>
       <CreateOfficeName>John</CreateOfficeName>
       <CreateOfficeCode>
            <OperatorId>ve</OperatorId>
            <OfficeCode>1234</OfficeCode>
            <CountryCodeLength>0</CountryCodeLength>
            <AreaCodeLength>3</AreaCodeLength>
       </CreateOfficeCode>
 </CreateOfficeGroup>

Can you answer the questions as individuals? Thank you very much!

18
  • 5
    Don't use bash/string processing for this. Use a tool made for processing XML. Something like XSLT or a real XML parser and a script in a language that lets you walk the resulting document and remove elements. Commented Nov 4, 2014 at 13:04
  • 1
    @CharlesDuffy true story, there's no misery here :) but one more reason is that sed is widely available on unix systems, while such a tool must most likely be installed. Commented Nov 4, 2014 at 14:25
  • 1
    @GézaTörök, to be sure, but because sed doesn't know XML syntax, any attempt to modify XML with sed is doomed to be incorrect. It doesn't know what is or isn't in a CDATA section, it doesn't know what is or isn't in a comment... you simply cannot parse XML accurately with sed. Commented Nov 4, 2014 at 14:27
  • 2
    @thahgr: Do you have a modern Python interpreter? If so, that includes a proper XML parser. Commented Nov 4, 2014 at 14:42
  • 3
    Alternately, does your system have xsltproc? Commented Nov 4, 2014 at 14:44

3 Answers 3

10

XMLStarlet is a command-line XML processor. Doing what you want with it is a one-line operation (until the desired recursive behavior is added), and will work for all variants of XML syntax describing the same input:

The simple version:

xmlstarlet ed \
  -d '//*[not(./*) and (not(./text()) or normalize-space(./text())="")]' \
  input.xml

The fancy version:

strip_recursively() {
  local doc last_doc
  IFS= read -r -d '' doc 
  while :; do
    last_doc=$doc
    doc=$(xmlstarlet ed \
           -d '//*[not(./*) and (not(./text()) or normalize-space(./text())="")]' \
           /dev/stdin <<<"$last_doc")
    if [[ $doc = "$last_doc" ]]; then
      printf '%s\n' "$doc"
      return
    fi
  done
}
strip_recursively <input.xml

/dev/stdin is used rather than - (at some cost to platform portability) for better portability across releases of XMLStarlet; adjust to taste.


With a system having only older dependencies installed, a more likely XML parser to have installed is that bundled with Python.

#!/usr/bin/env python

import xml.etree.ElementTree as etree
import sys

doc = etree.parse(sys.stdin)
def prune(parent):
    ever_changed = False
    while True:
        changed = False
        for el in parent.getchildren():
            if len(el.getchildren()) == 0:
                if ((el.text is None or el.text.strip() == '') and
                    (el.tail is None or el.tail.strip() == '')):
                    parent.remove(el)
                    changed = True
            else:
                changed = changed or prune(el)
        ever_changed = changed or ever_changed
        if changed is False:
            return ever_changed

prune(doc.getroot())
print etree.tostring(doc.getroot())
4
sed '#n
1h;1!H
$ { x
:remtag
  s#\(\n* *\)*<\([^>]*>\)\( *\n*\)*</\2##g
  t remtag

  p
  }' YourFile

(posix version so --posix on GNU sed)

  • recursively remove empty tag from lower lever to upper one until no more empty tag occur.
  • Not a XML parser so something like <tag1 prop="<tag2></tag2>"> ... will remove the prop content also and any other thing like that that xml allow.
3
  • 1
    I'm rather strongly opposed to any use of non-syntax-aware tools for parsing/generation of structured content (just because someone doesn't expect getting the corner cases wrong to matter doesn't mean those corner cases can't be leveraged in a security exploit or won't cause someone to lose a day of productivity tracking down a bug in an almost-unrelated component that calls something using this one years later)... but even I need to give this a +1. Commented Nov 4, 2014 at 18:45
  • 1
    I agree but sometimes other tools than 'basic one' are not allowed (this is my case) and some non XML parser, used with the knowledge of the limitation is the only solution in a short term. For a long term, think of the adapted tools (but unhoppefully, often solution are temporary long term ...) Commented Nov 5, 2014 at 7:36
  • I had a few thousand errors importing an xml file that worked on other tools, and no idea how many problems there were. This quickly did the job, removed the clutter. I could quickly find the remaining problems and verify the original import tool. For my use case, your sed was a perfect find. Thanks! Commented Aug 16, 2018 at 23:35
3

You can do the following with sed:

sed -i ':a;N;$!ba;s/<\([^>]*\)>[ \t\n]*<\/\1>//g;s/\([\n][\t\n ]*[\n]\)/\n/g;' yourfile.xml

The script at the beginning (:l;N;$!bl) appends all the lines together to the pattern space by a loop (:a - declares label a; N - append next line to the pattern space; $!bl - branch to a if last line is not reached)

The pattern of the first substitution is built up like opening tag (<\([^>]*\)>) - optional whitespace ([ \t\n]*) - closing tag (<\/\1>). Note the escaped parens surrounding the pattern for tag name, whose contents can be referred to as \1 afterwards in the expression. This is how the closing tag matches the opening tag.

Finally, the second substitution (s/[\n][\n]*/\n/g) simply removes consecutive newlines.

3
  • 1
    This doesn't seem to work in a test here at all. It also doesn't appear to handle elements with attributes (assuming it worked in the first place). There aren't any in the input but that may not hold true for the actual data. This also doesn't appear to handle <tag>\n</tag> (despite the inclusion of \n in that whitespace class because sed is line-oriented. Commented Nov 4, 2014 at 14:26
  • 1
    Leaves blank lines now. Now do you see my point about using an XML tool for this? =) Commented Nov 4, 2014 at 14:37
  • Ok, let's say i'm convinced :) Commented Nov 4, 2014 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.