10

I'm surprised that I'm not able to match a German umlaut in a regexp. I tried several approaches, most involving setting locales, but up to now to no avail.

locale.setlocale(locale.LC_ALL, 'de_DE.UTF-8')
re.findall(r'\w+', 'abc def g\xfci jkl', re.L)
re.findall(r'\w+', 'abc def g\xc3\xbci jkl', re.L)
re.findall(r'\w+', 'abc def güi jkl', re.L)
re.findall(r'\w+', u'abc def güi jkl', re.L)

None of these versions matches the umlaut-u (ü) correctly with \w+. Also removing the re.L flag or prefixing the pattern string with u (to make it unicode) did not help me.

Any ideas? How is the flag re.L used correctly?

2 Answers 2

17

Have you tried to use the re.UNICODE flag, as described in the doc?

>>> re.findall(r'\w+', 'abc def güi jkl', re.UNICODE)
['abc', 'def', 'g\xc3\xbci', 'jkl']

A quick search points to this thread that gives some explanation:

re.LOCALE just passes the character to the underlying C library. It really only works on bytestrings which have 1 byte per character. UTF-8 encodes codepoints outside the ASCII range to multiple bytes per codepoint, and the re module will treat each of those bytes as a separate character.

4
  • Nope, I didn't (mea culpa), I wasn't aware of the existence of that flag, and it solves my problem! Thank you, I'll flag this as the accepted answer soon (unless a more elaborate answer will also explain why my trials just using the locale flag did not work as expected.
    – Alfe
    Sep 2, 2012 at 22:50
  • If you're using Python 2, use Unicode strings (u"..."). Making the script UTF-8 is a good idea. Forget about the LOCALE flag, think of it as being for legacy stuff only (when you have no other choice).
    – MRAB
    Sep 3, 2012 at 0:55
  • Yeah, I read that sentence before posting my question. Nevertheless it is strange that the "underlying C library" is so hard to be nudged into accepting the umlaut as a word-character (\w). But, however, this seems to push the question away from Python and towards the C library which really is not my concern, as far as I have the solution with the re.U. Thanks for that!
    – Alfe
    Sep 3, 2012 at 7:05
  • I think this stinking re.LOCALE flag should be removed from the documentation and never mentioned. It is a fake, hypocrisy and a trap for the person coding in a hurry. Nov 26, 2013 at 14:33
0

In my case \S gave me better results than \w, plus saving the file as utf-8, plus using re.UNICODE

1
  • 1
    Maybe in your cases, but \S in general also matches things like punctuation and special characters (e. g. ♯ or → or similar). Not what I needed in my case.
    – Alfe
    Apr 6, 2013 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.