51

I've discovered that cElementTree is about 30 times faster than xml.dom.minidom and I'm rewriting my XML encoding/decoding code. However, I need to output XML that contains CDATA sections and there doesn't seem to be a way to do that with ElementTree.

Can it be done?

3
  • > I need to output XML that contains CDATA sections Why? It seems a strange requirment.
    – bortzmeyer
    Oct 15, 2008 at 12:14
  • 3
    It's a requirement I have - chunks of CDATA are sometimes much more human-readable.
    – grifaton
    Sep 6, 2010 at 22:39
  • @bortzmeyer It's useful for adding HTML to KML (Google Maps XML files).
    – logic-unit
    Jun 23, 2016 at 11:59

16 Answers 16

30

After a bit of work, I found the answer myself. Looking at the ElementTree.py source code, I found there was special handling of XML comments and preprocessing instructions. What they do is create a factory function for the special element type that uses a special (non-string) tag value to differentiate it from regular elements.

def Comment(text=None):
    element = Element(Comment)
    element.text = text
    return element

Then in the _write function of ElementTree that actually outputs the XML, there's a special case handling for comments:

if tag is Comment:
    file.write("<!-- %s -->" % _escape_cdata(node.text, encoding))

In order to support CDATA sections, I create a factory function called CDATA, extended the ElementTree class and changed the _write function to handle the CDATA elements.

This still doesn't help if you want to parse an XML with CDATA sections and then output it again with the CDATA sections, but it at least allows you to create XMLs with CDATA sections programmatically, which is what I needed to do.

The implementation seems to work with both ElementTree and cElementTree.

import elementtree.ElementTree as etree
#~ import cElementTree as etree

def CDATA(text=None):
    element = etree.Element(CDATA)
    element.text = text
    return element

class ElementTreeCDATA(etree.ElementTree):
    def _write(self, file, node, encoding, namespaces):
        if node.tag is CDATA:
            text = node.text.encode(encoding)
            file.write("\n<![CDATA[%s]]>\n" % text)
        else:
            etree.ElementTree._write(self, file, node, encoding, namespaces)

if __name__ == "__main__":
    import sys

    text = """
    <?xml version='1.0' encoding='utf-8'?>
    <text>
    This is just some sample text.
    </text>
    """

    e = etree.Element("data")
    cdata = CDATA(text)
    e.append(cdata)
    et = ElementTreeCDATA(e)
    et.write(sys.stdout, "utf-8")
4
  • 1
    This does not seem possible anymore since the write method is not there, and the _serialize* functions are static
    – Treviño
    Dec 30, 2012 at 15:05
  • What should I do since I can't use _write? So that means I can't use xml.elementtree? This is terrible.
    – elwc
    Jan 2, 2013 at 6:17
  • 3
    Thsio reciep won't work for Python 2.7 or 3.2 (and 3.3) - check @amaury's answer bellow. BAsically, teh new ElementTree does not have a "_write" method that can be overriden anymore.
    – jsbueno
    Jan 23, 2013 at 17:00
  • 1
    There is a CDATA element for etree you can use directly. lxml.de/api/lxml.etree.CDATA-class.html
    – coderek
    Oct 1, 2015 at 20:20
21

lxml has support for CDATA and API like ElementTree.

2
  • 2
    This is huge from the "don't roll your own XML parser" perspective. Mar 12, 2018 at 16:59
  • @iny I think your lxml link is broken. Nov 8, 2019 at 22:22
13

Here is a variant of gooli's solution that works for python 3.2:

import xml.etree.ElementTree as etree

def CDATA(text=None):
    element = etree.Element('![CDATA[')
    element.text = text
    return element

etree._original_serialize_xml = etree._serialize_xml
def _serialize_xml(write, elem, qnames, namespaces):
    if elem.tag == '![CDATA[':
        write("\n<%s%s]]>\n" % (
                elem.tag, elem.text))
        return
    return etree._original_serialize_xml(
        write, elem, qnames, namespaces)
etree._serialize_xml = etree._serialize['xml'] = _serialize_xml


if __name__ == "__main__":
    import sys

    text = """
    <?xml version='1.0' encoding='utf-8'?>
    <text>
    This is just some sample text.
    </text>
    """

    e = etree.Element("data")
    cdata = CDATA(text)
    e.append(cdata)
    et = etree.ElementTree(e)
    et.write(sys.stdout.buffer.raw, "utf-8")
3
  • This shoudl work fro Python 2.7 as well - as the original recipe does not. I jsut came up with another thing that is mode complicated than this.
    – jsbueno
    Jan 23, 2013 at 16:59
  • 1
    This needs updating to add the coding kwarg to the _serialize_xml def
    – Patrick
    Sep 28, 2014 at 17:58
  • 3
    for python 2.7 add an encoding arg to the serialize signature. change def _serialize_xml(write, elem, qnames, namespaces): to def _serialize_xml(write, elem, encoding, qnames, namespaces): change write, elem, qnames, namespaces) to write, elem, encoding, qnames, namespaces) change et.write(sys.stdout.buffer.raw, "utf-8") to et.write(sys.stdout, "utf-8")
    – Kevin
    Jan 15, 2015 at 5:53
12

Solution:

import xml.etree.ElementTree as ElementTree

def CDATA(text=None):
    element = ElementTree.Element('![CDATA[')
    element.text = text
    return element

ElementTree._original_serialize_xml = ElementTree._serialize_xml
def _serialize_xml(write, elem, qnames, namespaces,short_empty_elements, **kwargs):
    if elem.tag == '![CDATA[':
        write("\n<{}{}]]>\n".format(elem.tag, elem.text))
        if elem.tail:
            write(_escape_cdata(elem.tail))
    else:
        return ElementTree._original_serialize_xml(write, elem, qnames, namespaces,short_empty_elements, **kwargs)

ElementTree._serialize_xml = ElementTree._serialize['xml'] = _serialize_xml

if __name__ == "__main__":
    import sys

text = """
<?xml version='1.0' encoding='utf-8'?>
<text>
This is just some sample text.
</text>
"""

e = ElementTree.Element("data")
cdata = CDATA(text)
root.append(cdata)

Background:

I don't know whether previous versions of proposed code worked very well and whether ElementTree module has been updated but I have faced problems with using this trick:

etree._original_serialize_xml = etree._serialize_xml
def _serialize_xml(write, elem, qnames, namespaces):
    if elem.tag == '![CDATA[':
        write("\n<%s%s]]>\n" % (
                elem.tag, elem.text))
        return
    return etree._original_serialize_xml(
        write, elem, qnames, namespaces)
etree._serialize_xml = etree._serialize['xml'] = _serialize_xml

The problem with this approach is that after passing this exception, serializer is again treating it as normal tag afterwards. I was getting something like:

<textContent>
<![CDATA[this was the code I wanted to put inside of CDATA]]>
<![CDATA[>this was the code I wanted to put inside of CDATA</![CDATA[>
</textContent>

And of course we know that will cause only plenty of errors. Why that was happening though?

The answer is in this little guy:

return etree._original_serialize_xml(write, elem, qnames, namespaces)

We don't want to examine code once again through original serialise function if we have trapped our CDATA and successfully passed it through. Therefore in the "if" block we have to return original serialize function only when CDATA was not there. We were missing "else" before returning original function.

Moreover in my version ElementTree module, serialize function was desperately asking for "short_empty_element" argument. So the most recent version I would recommend looks like this(also with "tail"):

from xml.etree import ElementTree
from xml import etree

#in order to test it you have to create testing.xml file in the folder with the script
xmlParsedWithET = ElementTree.parse("testing.xml")
root = xmlParsedWithET.getroot()

def CDATA(text=None):
    element = ElementTree.Element('![CDATA[')
    element.text = text
    return element

ElementTree._original_serialize_xml = ElementTree._serialize_xml

def _serialize_xml(write, elem, qnames, namespaces,short_empty_elements, **kwargs):

    if elem.tag == '![CDATA[':
        write("\n<{}{}]]>\n".format(elem.tag, elem.text))
        if elem.tail:
            write(_escape_cdata(elem.tail))
    else:
        return ElementTree._original_serialize_xml(write, elem, qnames, namespaces,short_empty_elements, **kwargs)

ElementTree._serialize_xml = ElementTree._serialize['xml'] = _serialize_xml


text = """
<?xml version='1.0' encoding='utf-8'?>
<text>
This is just some sample text.
</text>
"""
e = ElementTree.Element("data")
cdata = CDATA(text)
root.append(cdata)

#tests
print(root)
print(root.getchildren()[0])
print(root.getchildren()[0].text + "\n\nyay!")

The output I got was:

<Element 'Database' at 0x10062e228>
<Element '![CDATA[' at 0x1021cc9a8>

<?xml version='1.0' encoding='utf-8'?>
<text>
This is just some sample text.
</text>


yay!

I wish you the same result!

4
  • 1
    Thank you! Your solution works great for me in Python 3.4.3, and it's really interesting that you only posted it yesterday, and I need it today. Haven't tested in 3.5, but I guess it will break sooner or later still, probably in the next version. Sigh.
    – 4ae1e1
    May 5, 2015 at 4:33
  • You are welcome. Please keep in mind that always while using ElementTree.parse, you will display only CDATA content (without cdata tag). In my code: 'xmlParsedWithET = ElementTree.parse("testing.xml")'. I figured out how by modifying the code just a little, using lxml you can preserve our precious cdata tags. Let me know if you are interested in that or only standard libs are ok for you
    – Kamil
    May 7, 2015 at 4:59
  • I was writing a generator for my blog and had to assemble an Atom 1.0 feed. This is kind of a one-off task (if it breaks in the future, I can always use a 3.4 virtualenv), so a hack on STL is acceptable to me.
    – 4ae1e1
    May 7, 2015 at 5:02
  • Thanks! Works well for python 3.6!
    – egvo
    Mar 3, 2020 at 12:03
7

It's not possible AFAIK... which is a pity. Basically, ElementTree modules assume that the reader is 100% XML compliant, so it shouldn't matter if they output a section as CDATA or some other format that generates the equivalent text.

See this thread on the Python mailing list for more info. Basically, they recommend some kind of DOM-based XML library instead.

4
  • 3
    I would not call it "a pity". For the XML infoset (the content), there is no difference between "<![CDATA[ & ]]>" and "&amp;"... Most XML parsers won't even let you know what was in the original document.
    – bortzmeyer
    Oct 15, 2008 at 12:13
  • 1
    That's true, but some data can be dumped and parsed much more efficiently in CDATA format. So it's a pain to not be able to tell an XML library to handle it in this way.
    – Dan Lenski
    Oct 15, 2008 at 21:02
  • 1
    The link seems like not available now.
    – Rahul K P
    Mar 30, 2016 at 11:26
  • Thanks. Replaced with Wayback Machine link.
    – Dan Lenski
    May 7, 2020 at 19:09
6

Actually this code has a bug, since you don't catch ]]> appearing in the data you are inserting as CDATA

as per Is there a way to escape a CDATA end token in xml?

you should break it into two CDATA's in that case, splitting the ]]> between the two.

basically data = data.replace("]]>", "]]]]><![CDATA[>")
(not necessarily correct, please verify)

5

You can override ElementTree _escape_cdata function:

import xml.etree.ElementTree as ET

def _escape_cdata(text, encoding):
    try:
        if "&" in text:
            text = text.replace("&", "&amp;")
        # if "<" in text:
            # text = text.replace("<", "&lt;")
        # if ">" in text:
            # text = text.replace(">", "&gt;")
        return text
    except TypeError:
        raise TypeError(
            "cannot serialize %r (type %s)" % (text, type(text).__name__)
        )

ET._escape_cdata = _escape_cdata

Note that you may not need pass extra encoding param, depending on your library/python version.

Now you can write CDATA into obj.text like:

root = ET.Element('root')
body = ET.SubElement(root, 'body')
body.text = '<![CDATA[perform extra angle brackets escape for this text]]>'
print(ET.tostring(root))

and get clear CDATA node:

<root>
    <body>
        <![CDATA[perform extra angle brackets escape for this text]]>
    </body>
</root>
3
  • How exactly do I use this to output CDATA sections? What is "contrib version"?
    – mzjn
    Oct 15, 2019 at 11:34
  • @mzjn thanks, edited. You can use it like usually do with inserting text to ET object. I mean obj.text='<![CDATA[text]]>'. "contrib version" is a library version or a specific python version library (not sure exactly where difference of args num is) Oct 15, 2019 at 18:20
  • 1
    Instead of commenting out those lines, I would just add if text.startswith("<![CDATA[") and text.endswith("]]>"): return text as the first line. That way you don't mess up non-cdata entries Apr 17, 2020 at 19:59
4

This ended up working for me in Python 2.7. Similar to Amaury's answer.

import xml.etree.ElementTree as ET

ET._original_serialize_xml = ET._serialize_xml


def _serialize_xml(write, elem, encoding, qnames, namespaces):
    if elem.tag == '![CDATA[':
        write("<%s%s]]>%s" % (elem.tag, elem.text, elem.tail))
        return
    return ET._original_serialize_xml(
         write, elem, encoding, qnames, namespaces)
ET._serialize_xml = ET._serialize['xml'] = _serialize_xml
3

for python3 and ElementTree you can use next reciept

import xml.etree.ElementTree as ET

ET._original_serialize_xml = ET._serialize_xml


def serialize_xml_with_CDATA(write, elem, qnames, namespaces, short_empty_elements, **kwargs):
    if elem.tag == 'CDATA':
        write("<![CDATA[{}]]>".format(elem.text))
        return
    return ET._original_serialize_xml(write, elem, qnames, namespaces, short_empty_elements, **kwargs)


ET._serialize_xml = ET._serialize['xml'] = serialize_xml_with_CDATA


def CDATA(text):
   element =  ET.Element("CDATA")
   element.text = text
   return element


my_xml = ET.Element("my_name")
my_xml.append(CDATA("<p>some text</p>")

tree = ElementTree(my_xml)

if you need xml as str, you can use

ET.tostring(tree)

or next hack (which almost same as code inside tostring())

fake_file = BytesIO()
tree.write(fake_file, encoding="utf-8", xml_declaration=True)
result_xml_text = str(fake_file.getvalue(), encoding="utf-8")

and get result

<?xml version='1.0' encoding='utf-8'?>
<my_name>
  <![CDATA[<p>some text</p>]]>
</my_name>
2

I've discovered a hack to get CDATA to work using comments:

node.append(etree.Comment(' --><![CDATA[' + data.replace(']]>', ']]]]><![CDATA[>') + ']]><!-- '))
1

The DOM has (atleast in level 2) an interface DATASection, and an operation Document::createCDATASection. They are extension interfaces, supported only if an implementation supports the "xml" feature.

from xml.dom import minidom

my_xmldoc=minidom.parse(xmlfile)

my_xmldoc.createCDATASection(data)

now u have cadata node add it wherever u want....

1

The accepted solution cannot work with Python 2.7. However, there is another package called lxml which (though slightly slower) shared a largely identical syntax with the xml.etree.ElementTree. lxml is able to both write and parse CDATA. Documentation here

0

Here's my version which is based on both gooli's and amaury's answers above. It works for both ElementTree 1.2.6 and 1.3.0, which use very different methods of doing this.

Note that gooli's does not work with 1.3.0, which seems to be the current standard in Python 2.7.x.

Also note that this version does not use the CDATA() method gooli used either.

import xml.etree.cElementTree as ET

class ElementTreeCDATA(ET.ElementTree):
    """Subclass of ElementTree which handles CDATA blocks reasonably"""

    def _write(self, file, node, encoding, namespaces):
        """This method is for ElementTree <= 1.2.6"""

        if node.tag == '![CDATA[':
            text = node.text.encode(encoding)
            file.write("\n<![CDATA[%s]]>\n" % text)
        else:
            ET.ElementTree._write(self, file, node, encoding, namespaces)

    def _serialize_xml(write, elem, qnames, namespaces):
        """This method is for ElementTree >= 1.3.0"""

        if elem.tag == '![CDATA[':
            write("\n<![CDATA[%s]]>\n" % elem.text)
        else:
            ET._serialize_xml(write, elem, qnames, namespaces)
0

I got here looking for a way to "parse an XML with CDATA sections and then output it again with the CDATA sections".

I was able to do this (maybe lxml has been updated since this post?) with the following: (it is a little rough - sorry ;-). Someone else may have a better way to find the CDATA sections programatically but I was too lazy.

 parser = etree.XMLParser(encoding='utf-8') # my original xml was utf-8 and that was a lot of the problem
 tree = etree.parse(ppath, parser)

 for cdat in tree.findall('./ProjectXMPMetadata'): # the tag where my CDATA lives
   cdat.text = etree.CDATA(cdat.text)

 # other stuff here

 tree.write(opath, encoding="UTF-8",)
0

Simple way of making .xml file with CDATA sections

The main idea is that we covert the element tree to a string and call unescape on it. Once we have the string we use standard python to write a string to a file.

Based on: How to write unescaped string to a XML element with ElementTree?

Code that generates the XML file

import xml.etree.ElementTree as ET
from xml.sax.saxutils import unescape

# defining the tree structure
element1 = ET.Element('test1')
element1.text = '<![CDATA[Wired & Forbidden]]>'

# & and <> are in a weird format
string1 = ET.tostring(element1).decode()
print(string1)

# now they are not weird anymore
# more formally, we unescape '&amp;', '&lt;', and '&gt;' in a string of data
# from https://docs.python.org/3.8/library/xml.sax.utils.html#xml.sax.saxutils.unescape
string1 = unescape(string1)
print(string1)

element2 = ET.Element('test2')
element2.text = '<![CDATA[Wired & Forbidden]]>'
string2 = unescape(ET.tostring(element2).decode())
print(string2)

# make the xml file and open in append mode
with open('foo.xml', 'a') as f:
    f.write(string1 + '\n')
    f.write(string2)

Output foo.xml

<test1><![CDATA[Wired & Forbidden]]></test1>
<test2><![CDATA[Wired & Forbidden]]></test2>
0

Combine with unescape https://wiki.python.org/moin/EscapingXml is very easy solution.

import xml.etree.cElementTree as ET
import xml.dom.minidom
from xml.sax.saxutils import unescape

m_encoding = 'UTF-8'

class Xml():

def generate(self, xmlstring):        
    root = ET.Element('info')
    ET.SubElement(root, "foo").text = "<![CDATA[{}]]>".format(xmlstring)
    
    dom = xml.dom.minidom.parseString(ET.tostring(root))
    xml_string = dom.toprettyxml()
    part1, part2 = xml_string.split('?>')
    
    with open(f"xmls/file.xml", 'w', encoding="UTF-8") as xfile:
        file_parts = part1 + 'encoding=\"{}\" standalone="yes"?>'.format(m_encoding) + part2
        xfile.write(unescape(file_parts, {"&apos;": "'", "&quot;": '"'}))
        xfile.close()
                

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.