4

I'm trying to render my sitemap.xml using flask, however I cannot seem to clean my input such that the rendering can work. The error is the following:

error on line 23325 at column 83: PCDATA invalid Char value 11

The code I'm using is:

url = prepend + "/explore/"+str(result['id'])+"/"+result['title'].encode('utf-8', 'xmlcharrefreplace').decode('utf-8')

I've also tried:

url = prepend + "/explore/"+str(result['id'])+"/"+result['title'].encode('ascii', 'xmlcharrefreplace').decode('ascii')

Also removing what is Char 11, doesn't seem to work either.

result["title"] = result["title"].replace('', '')

Is there a smarter way that I can clean any spurious characters or problem solve more accurately?

I've tried to flag the problem. It looks like a linefeed or something along those lines. I've tried before and after the URL is built.

XX XXX XXX XXX  
                Step by step guide

but, I cannot find anything in the python arsenal to remove the linefeed. I'm trying, replace('\r', ''), strip() etc etc, but still this persists.

2

1 Answer 1

4

“Char value 11” (0xB in hex) refers to the vertical tab \v, not the carriage return \r, nor the line feed \n.

The vertical tab is not a valid character in an XML document. It’s so invalid that you can’t even represent it as a character reference like . You have to remove it from the string yourself:

result["title"] = result["title"].replace('\x0b', '')

While you’re at it, you may want to remove other invalid characters as well, as they have a tendency to pop up inscrutably in user-supplied data and bite you in production. I’m not aware of common library functions to do this, so I came up with my own function that makes both XML and HTML5 more or less happy:

import re

def printable(s):
    # Based on `XML 1.0 section 2.2 <https://www.w3.org/TR/xml/#charsets>`_,
    # with the addition of U+0085,
    # which the W3C (Nu) validator also marked as a "forbidden code point".
    # Even with this code, the validator still complains about
    # "Text run is not in Unicode Normalization Form C"
    # and "Document uses the Unicode Private Use Area(s)".
    return re.sub(
        pattern=(u'[\u0000-\u0008\u000B\u000C\u000E-\u001F'
                 u'\u007F-\u009F\uD800-\uDFFF\uFDD0-\uFDEF\uFFFE\uFFFF]'),
        repl=u'\N{REPLACEMENT CHARACTER}',
        string=s
    )

As for xmlcharrefreplace, that’s not what you’re looking for at all. It’s not an escaping mechanism for XML-invalid characters. It’s a way to preserve characters that are outside of the chosen encoding. It would make sense in a situation like this:

>>> print(u'Liberté, égalité, fraternité!'.encode('ascii', 'xmlcharrefreplace'))
Libert&#233;, &#233;galit&#233;, fraternit&#233;!

Not the answer you're looking for? Browse other questions tagged or ask your own question.