8

look at the following snippet:

>>> import unicodedata
>>> from unicodedata import normalize, name

>>> normalize('NFKD', u'\xb4')
u' \u0301'

>>> normalize('NFKD', u'a\xb4a')
u'a \u0301a'

>>> normalize('NFKC', u'a\xb4a')
u'a \u0301a'

>>> name(u'\xb4'), name(u'\u0301')
('ACUTE ACCENT', 'COMBINING ACUTE ACCENT')

I am trying to understand if the behavior to translate u'\xb4' to u' \u0301' is correct. Why does it pad the combining acute accent with a space? Why does it translate the u\xb4 at all?

At fileformat we see that the ACUTE ACCENT used to be called SPACING ACUTE. I thought, it just meant that the cursor should move instead of waiting for the following character to be typed in.

UPD: in case someone is interested, here is a list if unicode characters that after NFKC normalization have a space in the beginning: http://pastebin.com/Z99r5AK9

3 Answers 3

11

An accent character is the combination of a space and a combining accent character, as specified in the Unicode standard:

>>> import unicodedata
>>> unicodedata.decomposition(u'\xb4')
'<compat> 0020 0301'

The \u00B4 character has a somewhat ambiguous history, but the Unicode standard has decided to treat it as whitespace + accent, even though it has often been used as just a diacritic mark, see this discussion.

You could perhaps use \u02CA as an alternative; it is not treated as whitespace, and has no decomposition specified. It is instead qualified as a letter, so your mileage may vary.

2
  • U+00B4 ACUTE ACCENT and U+02CA MODIFIER LETTER ACUTE ACCENT are distinct characters and should not be expected to be “rendered the same”. The former is far more often present in fonts. When both are present, they usually look similar, or the same, but they have different properties defined in the Unicode Standard (e.g., general category Symbol, Modifier vs. Letter, Modifier). And, for example, in the Doulos SIL font, their glyphs are apparently identical but advance widths are different; and in MingLiU, the slopes are very different. Dec 19, 2012 at 18:08
  • @JukkaK.Korpela: I stand corrected; I must have misunderstood the context, I'll remove that part.
    – Martijn Pieters
    Dec 19, 2012 at 19:46
4

Take a look at the Unicode Collation Algorithm document. In particular, note that

Compatibility normalization (NFKC) folds stand-alone accents to a combination of space + combining accent.

0
3

In NFKD, accented characters are stored in a "split" way: first the character to be accented and then the combining accent: u' \u0301'

In NFKC, accented characters are stored in a "combined" way: there is a dedicated Unicode code point: u'\xb4', which is shorthand for u'\u00b4'.

Both of them represent the accent alone, which can be seen as an accent over a space character.

1
  • 1
    it is also interesting why normalize('NFKC', u' \u0301') does not return u'\xb4' though.
    – newtover
    Dec 19, 2012 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.