10

Do Arabic characters have different Unicode code points based on position in string, or is it a visual solution?

This is the same word, 3 times, with white spaces and without seems like it's the same Unicode value.

عربى‎
عرب ى‎
ع ربى‎

What I need to do, is to scan a list of Arabic strings, and get their values. Using those values, I will choose the icon of the specific letter to display. However, if it's the same code point, the meaning is that I need to create a logic of my own in the code, which I want to avoid.

3
  • 3
    Please read the Wikipedia entry on Arabic in Unicode. The answer is: possibly. Isolated shapes may get rendered differently by text rendering engines and/or by OpenType instructions in the font file itself.
    – Jongware
    Commented Nov 15, 2015 at 11:21
  • 2
    The answer is “yes”. In Arabic script the letters change shape depending on the surrounding letters. And the different shapes of a letter are represented by the same code point.
    – roeland
    Commented Nov 15, 2015 at 21:22
  • @roeland: Except it's not that simple, because Unicode also provides code points for initil, middle and final forms of letters.
    – Flimm
    Commented Jan 27, 2016 at 11:35

2 Answers 2

15

Different shapes have different unicode, for example the letter ت \u062A has all those codes for different shapes: \uFE95 ﺕ , \uFE97 ﺗ, \uFE98 ﺘ , \uFE96 ﺖ.

Though, mostly, Arabic texts are kept with the main unshaped unicode. shaped forms are used only in rendering. so if you check your text through a program, you will find it mostly unshaped.

If you want all letters shaped, you may use a reshaper library like: Python Arabic Reshaper:

   import arabic_reshaper
   reshaped_text = arabic_reshaper.reshape(u'اللغة العربية رائعة')

If you want all letters unshaped, use the shaping map down to convert letters to their main shape.

Here is a shaping map:

SHAPING = {
 u'\u0621' : ( u'\uFE80' ) ,
 u'\u0622' : ( u'\uFE81', u'\uFE82' ) ,
 u'\u0623' : ( u'\uFE83', u'\uFE84' ) ,
 u'\u0624' : ( u'\uFE85' , u'\uFE86' ) ,
 u'\u0625' : ( u'\uFE87' , u'\uFE88' ) ,
 u'\u0626' : ( u'\uFE89' , u'\uFE8B' , u'\uFE8C' , u'\uFE8A' ) ,
 u'\u0627' : ( u'\uFE8D' , u'\uFE8E' ) ,
 u'\u0628' : ( u'\uFE8F' , u'\uFE91' , u'\uFE92' , u'\uFE90' ) ,
 u'\u0629' : ( u'\uFE93' , u'\uFE94' ) ,
 u'\u062A' : ( u'\uFE95' , u'\uFE97' , u'\uFE98' , u'\uFE96' ) ,
 u'\u062B' : ( u'\uFE99' , u'\uFE9B' , u'\uFE9C' , u'\uFE9A' ) ,
 u'\u062C' : ( u'\uFE9D' , u'\uFE9F' , u'\uFEA0', u'\uFE9E' ) ,
 u'\u062D' : ( u'\uFEA1' , u'\uFEA3' , u'\uFEA4' , u'\uFEA2' ) ,
 u'\u062E' : ( u'\uFEA5' , u'\uFEA7' , u'\uFEA8' , u'\uFEA6' ) ,
 u'\u062F' : ( u'\uFEA9' , u'\uFEAA' ) ,
 u'\u0630' : ( u'\uFEAB'  , u'\uFEAC' ) ,
 u'\u0631' : ( u'\uFEAD' , u'\uFEAE' ) ,
 u'\u0632' : ( u'\uFEAF'  , u'\uFEB0' ) ,
 u'\u0633' : ( u'\uFEB1' , u'\uFEB3' , u'\uFEB4' , u'\uFEB2' ) ,
 u'\u0634' : ( u'\uFEB5' , u'\uFEB7' , u'\uFEB8' , u'\uFEB6' ) ,
 u'\u0635' : ( u'\uFEB9' , u'\uFEBB' , u'\uFEBC' , u'\uFEBA' ) ,
 u'\u0636' : ( u'\uFEBD' , u'\uFEBF' , u'\uFEC0' , u'\uFEBE' ) ,
 u'\u0637' : ( u'\uFEC1' , u'\uFEC3' , u'\uFEC4' , u'\uFEC2' ) ,
 u'\u0638' : ( u'\uFEC5' , u'\uFEC7' , u'\uFEC8' , u'\uFEC6' ) ,
 u'\u0639' : ( u'\uFEC9' , u'\uFECB' , u'\uFECC' , u'\uFECA' ) ,
 u'\u063A' : ( u'\uFECD' , u'\uFECF' , u'\uFED0', u'\uFECE' ) ,
 u'\u0640' : ( u'\u0640' ) ,
 u'\u0641' : ( u'\uFED1' , u'\uFED3' , u'\uFED4' , u'\uFED2' ) ,
 u'\u0642' : ( u'\uFED5' , u'\uFED7' , u'\uFED8' , u'\uFED6' ) ,
 u'\u0643' : ( u'\uFED9' , u'\uFEDB' , u'\uFEDC' , u'\uFEDA' ) ,
 u'\u0644' : ( u'\uFEDD' , u'\uFEDF' , u'\uFEE0', u'\uFEDE' ) ,
 u'\u0645' : ( u'\uFEE1' , u'\uFEE3' , u'\uFEE4' , u'\uFEE2' ) ,
 u'\u0646' : ( u'\uFEE5' , u'\uFEE7' , u'\uFEE8' , u'\uFEE6' ) ,
 u'\u0647' : ( u'\uFEE9' , u'\uFEEB' , u'\uFEEC' , u'\uFEEA' ) ,
 u'\u0648' : ( u'\uFEED' , u'\uFEEE' ) ,
 u'\u0649' : ( u'\uFEEF' , u'\uFEF0' ) ,
 u'\u064A' : ( u'\uFEF1' , u'\uFEF3' , u'\uFEF4' , u'\uFEF2' )
}
4
  • Is there any arabic reshaper in JAVA ?
    – Mneckoee
    Commented Jan 6, 2018 at 5:39
  • 2
    How about arabic reshaper in c#?
    – Mneckoee
    Commented Jan 30, 2018 at 5:19
  • @BigOther, Apologies for the direct approach. I noticed you were involved in a previous Arabic SE site proposal. So I thought you may be interested to support a new one. Please spread the word to get wider support: area51.stackexchange.com/proposals/123866/… Commented Feb 8, 2020 at 13:29
  • 1
    To convert shaped characters to unshaped characters, (or to use different terminology, general charaters to presentational characters), you can use unicodedata.normalize("NFKC", string) or unicodedata.normalize.("NFKD", string) as well.
    – Flimm
    Commented Apr 20, 2020 at 12:37
5

Arabic has 5 blocks of characters reserved for it in Unicode:

  • U+0600 .. U+06FF Arabic
  • U+0750 .. U+077F Arabic Supplement
  • U+08A0 .. U+08FF Arabic Extended A
  • U+FB50 .. U+FDFF Arabic Presentation Forms A
  • U+FE70 .. U+FEFF Arabic Presentation Forms B

The example text in the question is all encoded with the 4 code points:

  • UTF-8 0xD8 0xB9 = U+0639 = ARABIC LETTER AIN
  • UTF-8 0xD8 0xB1 = U+0631 = ARABIC LETTER REH
  • UTF-8 0xD8 0xA8 = U+0628 = ARABIC LETTER BEH
  • UTF-8 0xD9 0x89 = U+0649 = ARABIC LETTER ALEF MAKSURA

Additionally, there are spaces and some occurrences of:

  • UTF-8 0xE2 0x80 0x8E = U+200E = LEFT-TO-RIGHT MARK (LRM)

The fact that the Arabic letters are displayed differently despite the same Unicode code point being used to store the data shows that you will need to adapt the glyph that is displayed to its position relative to other characters (beginning, middle, end of word, or standalone). You can read Chapter 9 (Middle East–I) to find out a lot more about the handling of Arabic text.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.