15

I am attempting to use the re module in Python 2.7.3 with Unicode encoded Devnagari text. I have added from __future__ import unicode_literals to the top of my code so all strings literals should be unicode objects.

However, I am running into some odd problems with Python's regex matching. For instance, consider this name: "किशोरी". This is a (mis-spelled) name, in Hindi, entered by one of my users. Any Hindi reader would recognise this as a word.

The following returns a match, as it should:

re.search("^[\w\s][\w\s]*","किशोरी",re.UNICODE)

But this does not:

re.search("^[\w\s][\w\s]*$","किशोरी",re.UNICODE)

Some spelunking revealed that only one character in this string, character 0915 (क), is recognised as falling within the \w character class. This is incorrect, as the Unicode Character Database file on "derived core properties" lists other characters (I have not checked all) in this string as alphabetic ones - as indeed they are.

Is this just a bug in Python's implementation? I could get around this by manually defining all the Devnagari alphanumeric characters as a character range, but that would be painful. Or am I doing something wrong?

3 Answers 3

15

It is a bug in the re module and it is fixed in the regex module:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import unicodedata
import re
import regex  # $ pip install regex

word = "किशोरी"


def test(re_):
    assert re_.search("^\\w+$", word, flags=re_.UNICODE)

print([unicodedata.category(cp) for cp in word])
print(" ".join(ch for ch in regex.findall("\\X", word)))
assert all(regex.match("\\w$", c) for c in ["a", "\u093f", "\u0915"])

test(regex)
test(re)  # fails

The output shows that there are 6 codepoints in "किशोरी", but only 3 user-perceived characters (extended grapheme clusters). It would be wrong to break a word inside a character. Unicode Text Segmentation says:

Word boundaries, line boundaries, and sentence boundaries should not occur within a grapheme cluster: in other words, a grapheme cluster should be an atomic unit with respect to the process of determining these other boundaries.

here and further emphasis is mine

A word boundary \b is defined as a transition from \w to \W (or in reverse) in the docs:

Note that formally, \b is defined as the boundary between a \w and a \W character (or vice versa), or between \w and the beginning/end of the string, ...

Therefore either all codepoints that form a single character are \w or they are all \W. In this case "किशोरी" matches ^\w{6}$.


From the docs for \w in Python 2:

If UNICODE is set, this will match the characters [0-9_] plus whatever is classified as alphanumeric in the Unicode character properties database.

in Python 3:

Matches Unicode word characters; this includes most characters that can be part of a word in any language, as well as numbers and the underscore.

From regex docs:

Definition of 'word' character (issue #1693050):

The definition of a 'word' character has been expanded for Unicode. It now conforms to the Unicode specification at http://www.unicode.org/reports/tr29/. This applies to \w, \W, \b and \B.

According to unicode.org U+093F (DEVANAGARI VOWEL SIGN I) is alnum and alphabetic so regex is also correct to consider it \w even if we follow definitions that are not based on word boundaries.

3
  • Yes, can confirm that regex module works. The [[:alnum:]] POSIX character class also works with the regex module.
    – ShankarG
    Commented Oct 5, 2012 at 14:28
  • @ShankarG: perl agrees: echo किशोरी | perl -CS -ne'print if /^\w+$/' (assume utf-8 io).
    – jfs
    Commented Oct 5, 2012 at 15:30
  • Changed my "accepted" mark to this answer, since this is in fact the correct answer - it is indeed a bug in the re module.
    – ShankarG
    Commented Oct 6, 2012 at 3:59
2

From Character Map:

‍ि

U+093F DEVANAGARI VOWEL SIGN I

General Character Properties

In Unicode since: 1.1 Unicode category: Mark, Spacing Combining

So, technically speaking this is not a letter and doesn't fall under \w even with re.UNICODE. You can try using regex with Unicode character properties instead in order to include these sorts of characters.

9
  • According to the list I linked to on derived code properties above 093F is indeed classified as an alphabetic character. Not sure what is going on. In any case, these have to be recognised as such - these characters never stand on their own, they are modifications of existing characters to indicate particular vowel sounds (in this case the character indicates that the subsequent "ka" should be pronounced as a "ki").
    – ShankarG
    Commented Oct 5, 2012 at 13:34
  • If one were to reject these and similar charcters as alphabetical characters, no Devnagari text in any of the languages that use it (Hindi, Bengali, Marathi, etc.) would ever be recognised as alphabetical in nature.
    – ShankarG
    Commented Oct 5, 2012 at 13:34
  • "093E..0940 ; Alphabetic # Mc [3] DEVANAGARI VOWEL SIGN AA..DEVANAGARI VOWEL SIGN II" Mc. Mc. Commented Oct 5, 2012 at 13:36
  • What does "Mc" mean? And doesn't it also say Alphabetic?
    – ShankarG
    Commented Oct 5, 2012 at 13:36
  • It's alphabetic, but it's not a letter. Commented Oct 5, 2012 at 13:37
2

I tested the following:

import unicodedata
for c in "किशोरी":
    print unicodedata.category(c)
    print unicodedata.name(c)

which displays in my case:

Lo
DEVANAGARI LETTER KA
Mc
DEVANAGARI VOWEL SIGN I
Lo
DEVANAGARI LETTER SHA
Mc
DEVANAGARI VOWEL SIGN O
Lo
DEVANAGARI LETTER RA
Mc
DEVANAGARI VOWEL SIGN II

Unicode stuff is hard to debug because copy and paste can mess up the data and I don't know hindi. But in some languages you can encode characters in different ways in unicode. Is it possible, that you have to normalize your string somehow before matching? To me it looks ok that a vowel sign is not matched by \w.

2
  • See comment on Ignacio's answer below. What do you mean by normalizing, though? Maybe that would do the trick.
    – ShankarG
    Commented Oct 5, 2012 at 13:35
  • I don't remember the exact details out of my head, but there a characters which exist on their own and can also be a cominbation. For example the german ä. It's a single character, but as far as I know there's a possiblity to encode it as a + marker for having those dots over it. And there's a transformation between both versions. Sorry, I have no possibility to check the details at the moment.
    – Achim
    Commented Oct 5, 2012 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.