10

I have a regex to detect invalid xml 1.0 characters in a unicode string:

bad_xml_chars = re.compile(u'[^\x09\x0A\x0D\u0020-\uD7FF\uE000-\uFFFD\U00010000-\U0010FFFF]', re.U)

On Linux/python2.7, this works perfectly. On windows the following is raised:

  File "C:\Python27\lib\re.py", line 190, in compile
    return _compile(pattern, flags)
  File "C:\Python27\lib\re.py", line 242, in _compile
    raise error, v # invalid expression
  sre_constants.error: bad character range

Any ideas why this isn't compiling on Windows?

2
  • 1
    I get the same error on Linux (also Python 2.7).
    – interjay
    Dec 13, 2012 at 17:38
  • Works here on Linux, Python 2.7 Dec 13, 2012 at 17:41

3 Answers 3

16

You have a narrow Python build on Windows, so Unicode uses UTF-16. This means that Unicode characters higher than \uFFFF will be two separate characters in the Python string. You should see something like this:

>>> len(u'\U00010000')
2
>>> u'\U00010000'[0]
u'\ud800'
>>> u'\U00010000'[1]
u'\udc00'

Here is how the regex engine will attempt to interpret your string on narrow builds:

[^\x09\x0A\x0D\u0020-\ud7ff\ue000-\ufffd\ud800\udc00-\udbff\udfff]

You can see here that \udc00-\udbff is where the invalid range message is coming from.

1
  • This should be fixed in Python 3.3 - all builds will behave like wide builds (except using less memory). I appreciate it may not be feasible for the OP to use 3.3.
    – Thomas K
    Dec 13, 2012 at 18:10
7

It doesn't work because the Windows version of Python uses 16 bits to represent unicode characters, encoded as UTF-16. Code points 10000 and above are represented as two code units in UTF-16, and this confuses the re range representation which expects a single character on either side of the -.

This is how the string you're passing to re.compile is split into characters:

>>> [x for x in u'[^\x09\x0A\x0D\u0020-\uD7FF\uE000-\uFFFD\U00010000-\U0010FFFF]']
[u'[', u'^', u'\t', u'\n', u'\r', u' ', u'-', u'\ud7ff', u'\ue000', u'-', 
 u'\ufffd', u'\ud800', u'\udc00', u'-', u'\udbff', u'\udfff', u']']

Note that \U00010000-\U0010FFFF is represented as 5 characters:

u'\ud800', u'\udc00', u'-', u'\udbff', u'\udfff'

Inside the character set [...], re.compile interprets this as the characters u'\ud800' and u'\udfff', and the range u'\udc00' - u'\udbff'. This range is invalid because its end is smaller than its start, which causes the error.

1

There's a section in the standard library that deals with bad character ranges (Lib/sre_compile.py:450):

if code1[0] != LITERAL or code2[0] != LITERAL:
    raise error, "bad character range"
lo = code1[1]
hi = code2[1]
if hi < lo:
    raise error, "bad character range"

When it compares the lo and hi literals of your \U00010000-\U0010FFFF character range, they come out to being ordinals 56320 and 56319, respectively (which of course fails, because the range appears to be backwards).

As others have said, this is because Python is treating your 8-character Unicode literals as two separate characters.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.