4

How can I find information about a Unicode character(e.g. character set it belongs to) in Java script ?

E.g.

00e9  LATIN SMALL LETTER E WITH ACUTE
0bf2  TAMIL NUMBER ONE THOUSAND

I am aware of a way to find details about a Unicode code point in python, using theunicodedata library. Is there a way to find out this information in JS?

PS: I am using this for chrome extension development, so a solution using their APIs is also good.

4
  • Can you guarantee that the input is always limited to the chosen language? For example naive is often spelt with a diaresis (ï), and annoying people like me use unicode greek like α. Consider just measuring the character set of the inputs you get.
    – Phil H
    May 9, 2012 at 11:22
  • I cannot guarantee that. The idea is to find out if the domain of a url entered in the browser has characters which are outside the languages currently set. E.g. if Chinese and English are set, I need to detect if a character which belongs to neither is part of the url. This can just be restricted to the alphabet of the language in this case.
    – varrunr
    May 10, 2012 at 4:17
  • Yes, you’re wrong about English being in the code point range of 0–127. Very very wrong.
    – tchrist
    May 13, 2012 at 14:38
  • Thanks for the clarifications. I have fixed the problem description after going through the answers.
    – varrunr
    May 13, 2012 at 22:15

3 Answers 3

4

English-language text is dominated by code points from the Latin, Common, and Inherited scripts, and in some corpora, also Greek.

For example, the PubMed Open Access collection, which is a very large collection of all English-language text, is filled with non-ASCII code points. Fully 90% of these are accounted for by only 36 distinct code points, as follows:

rank  percent cumulative  code glyph  GC=??   Name
---------------------------------------------------------------------
   1  18.553%  18.553%  U+02013 ‹–›  GC=Pd    EN DASH
   2   7.422%  25.974%  U+000A0 ‹ ›  GC=Zs    NO-BREAK SPACE
   3   7.033%  33.007%  U+000B1 ‹±›  GC=Sm    PLUS-MINUS SIGN
   4   5.461%  38.469%  U+02212 ‹−›  GC=Sm    MINUS SIGN
   5   4.196%  42.664%  U+02003 ‹ ›  GC=Zs    EM SPACE
   6   3.682%  46.346%  U+003BC ‹μ›  GC=Ll    GREEK SMALL LETTER MU
   7   3.619%  49.965%  U+003B2 ‹β›  GC=Ll    GREEK SMALL LETTER BETA
   8   3.568%  53.534%  U+003B1 ‹α›  GC=Ll    GREEK SMALL LETTER ALPHA
   9   3.426%  56.959%  U+0200A ‹ ›  GC=Zs    HAIR SPACE
  10   3.221%  60.181%  U+000B0 ‹°›  GC=So    DEGREE SIGN
  11   2.931%  63.112%  U+02009 ‹ ›  GC=Zs    THIN SPACE
  12   2.620%  65.732%  U+02019 ‹’›  GC=Pf    RIGHT SINGLE QUOTATION MARK
  13   2.506%  68.238%  U+02032 ‹′›  GC=Po    PRIME
  14   2.441%  70.679%  U+000D7 ‹×›  GC=Sm    MULTIPLICATION SIGN
  15   2.042%  72.722%  U+0201D ‹”›  GC=Pf    RIGHT DOUBLE QUOTATION MARK
  16   2.039%  74.761%  U+0201C ‹“›  GC=Pi    LEFT DOUBLE QUOTATION MARK
  17   1.536%  76.296%  U+00394 ‹Δ›  GC=Lu    GREEK CAPITAL LETTER DELTA
  18   1.415%  77.712%  U+000B5 ‹µ›  GC=Ll    MICRO SIGN
  19   1.337%  79.049%  U+003B3 ‹γ›  GC=Ll    GREEK SMALL LETTER GAMMA
  20   1.210%  80.259%  U+000E9 ‹é›  GC=Ll    LATIN SMALL LETTER E WITH ACUTE
  21   1.152%  81.410%  U+02014 ‹—›  GC=Pd    EM DASH
  22   1.135%  82.546%  U+02018 ‹‘›  GC=Pi    LEFT SINGLE QUOTATION MARK
  23   0.998%  83.543%  U+000A9 ‹©›  GC=So    COPYRIGHT SIGN
  24   0.710%  84.253%  U+02265 ‹≥›  GC=Sm    GREATER-THAN OR EQUAL TO
  25   0.600%  84.853%  U+000F6 ‹ö›  GC=Ll    LATIN SMALL LETTER O WITH DIAERESIS
  26   0.599%  85.452%  U+000B7 ‹·›  GC=Po    MIDDLE DOT
  27   0.597%  86.049%  U+02022 ‹•›  GC=Po    BULLET
  28   0.594%  86.644%  U+0223C ‹∼›  GC=Sm    TILDE OPERATOR
  29   0.573%  87.217%  U+003BA ‹κ›  GC=Ll    GREEK SMALL LETTER KAPPA
  30   0.569%  87.785%  U+000FC ‹ü›  GC=Ll    LATIN SMALL LETTER U WITH DIAERESIS
  31   0.493%  88.278%  U+02264 ‹≤›  GC=Sm    LESS-THAN OR EQUAL TO
  32   0.440%  88.718%  U+000AE ‹®›  GC=So    REGISTERED SIGN
  33   0.433%  89.152%  U+000E4 ‹ä›  GC=Ll    LATIN SMALL LETTER A WITH DIAERESIS
  34   0.422%  89.573%  U+02020 ‹†›  GC=Po    DAGGER
  35   0.407%  89.980%  U+003B4 ‹δ›  GC=Ll    GREEK SMALL LETTER DELTA

One way to detect those would be to use the Unicode regular expression that says a character must either be from the Latin, Greek, Common, or Inherited scripts.

In this corpus, the top four comprise will over 99% of the code points. However, there are also a great many super-low-frequency code points in this dataset that fall outside those four scripts (e.g. Cyrillic, Han, Kana, Hangul, etc.). You would throw those out as false negatives if you restricted input to the four ultra-common scripts previously listed. There are 239 such distinct code points in this dataset, of which the top 50 most frequent are the following:

rank  percent cumulative  code glyph  GC=??   Name
---------------------------------------------------------------------
 295   0.002%  99.828%  U+00424 ‹Ф›  GC=Lu    CYRILLIC CAPITAL LETTER EF
 381   0.001%  99.916%  U+0043A ‹к›  GC=Ll    CYRILLIC SMALL LETTER KA
 454   0.000%  99.949%  U+00413 ‹Г›  GC=Lu    CYRILLIC CAPITAL LETTER GHE
 491   0.000%  99.959%  U+0AD6D ‹국›  GC=Lo    HANGUL SYLLABLE GUG
 499   0.000%  99.961%  U+003EC ‹Ϭ›  GC=Lu    COPTIC CAPITAL LETTER SHIMA
 513   0.000%  99.965%  U+00406 ‹І›  GC=Lu    CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I
 528   0.000%  99.968%  U+00416 ‹Ж›  GC=Lu    CYRILLIC CAPITAL LETTER ZHE
 534   0.000%  99.969%  U+00430 ‹а›  GC=Ll    CYRILLIC SMALL LETTER A
 539   0.000%  99.970%  U+0041F ‹П›  GC=Lu    CYRILLIC CAPITAL LETTER PE
 545   0.000%  99.971%  U+00421 ‹С›  GC=Lu    CYRILLIC CAPITAL LETTER ES
 553   0.000%  99.972%  U+0D55C ‹한›  GC=Lo    HANGUL SYLLABLE HAN
 555   0.000%  99.972%  U+00404 ‹Є›  GC=Lu    CYRILLIC CAPITAL LETTER UKRAINIAN IE
 566   0.000%  99.974%  U+0C5B4 ‹어›  GC=Lo    HANGUL SYLLABLE EO
 567   0.000%  99.974%  U+0041A ‹К›  GC=Lu    CYRILLIC CAPITAL LETTER KA
 568   0.000%  99.974%  U+0041B ‹Л›  GC=Lu    CYRILLIC CAPITAL LETTER EL
 571   0.000%  99.975%  U+0B2C8 ‹니›  GC=Lo    HANGUL SYLLABLE NI
 575   0.000%  99.975%  U+0AE4C ‹까›  GC=Lo    HANGUL SYLLABLE GGA
 578   0.000%  99.976%  U+00428 ‹Ш›  GC=Lu    CYRILLIC CAPITAL LETTER SHA
 579   0.000%  99.976%  U+00454 ‹є›  GC=Ll    CYRILLIC SMALL LETTER UKRAINIAN IE
 585   0.000%  99.977%  U+00418 ‹И›  GC=Lu    CYRILLIC CAPITAL LETTER I
 587   0.000%  99.977%  U+0B2E4 ‹다›  GC=Lo    HANGUL SYLLABLE DA
 600   0.000%  99.978%  U+00440 ‹р›  GC=Ll    CYRILLIC SMALL LETTER ER
 610   0.000%  99.980%  U+00457 ‹ї›  GC=Ll    CYRILLIC SMALL LETTER YI
 614   0.000%  99.980%  U+0C74C ‹음›  GC=Lo    HANGUL SYLLABLE EUM
 623   0.000%  99.981%  U+0BD80 ‹부›  GC=Lo    HANGUL SYLLABLE BU
 624   0.000%  99.981%  U+0C545 ‹악›  GC=Lo    HANGUL SYLLABLE AG
 625   0.000%  99.981%  U+0C778 ‹인›  GC=Lo    HANGUL SYLLABLE IN
 640   0.000%  99.982%  U+0C5D0 ‹에›  GC=Lo    HANGUL SYLLABLE E
 641   0.000%  99.983%  U+0C744 ‹을›  GC=Lo    HANGUL SYLLABLE EUL
 645   0.000%  99.983%  U+00438 ‹и›  GC=Ll    CYRILLIC SMALL LETTER I
 664   0.000%  99.984%  U+0041C ‹М›  GC=Lu    CYRILLIC CAPITAL LETTER EM
 665   0.000%  99.984%  U+00436 ‹ж›  GC=Ll    CYRILLIC SMALL LETTER ZHE
 674   0.000%  99.985%  U+0C774 ‹이›  GC=Lo    HANGUL SYLLABLE I
 678   0.000%  99.985%  U+00431 ‹б›  GC=Ll    CYRILLIC SMALL LETTER BE
 679   0.000%  99.986%  U+00435 ‹е›  GC=Ll    CYRILLIC SMALL LETTER IE
 689   0.000%  99.986%  U+0B300 ‹대›  GC=Lo    HANGUL SYLLABLE DAE
 690   0.000%  99.986%  U+0BD84 ‹분›  GC=Lo    HANGUL SYLLABLE BUN
 691   0.000%  99.986%  U+0C678 ‹외›  GC=Lo    HANGUL SYLLABLE OE
 696   0.000%  99.987%  U+005DB ‹כ›  GC=Lo    HEBREW LETTER KAF
 703   0.000%  99.987%  U+0B85C ‹로›  GC=Lo    HANGUL SYLLABLE RO
 711   0.000%  99.988%  U+0041D ‹Н›  GC=Lu    CYRILLIC CAPITAL LETTER EN
 712   0.000%  99.988%  U+004D9 ‹ә›  GC=Ll    CYRILLIC SMALL LETTER SCHWA
 725   0.000%  99.988%  U+0B294 ‹는›  GC=Lo    HANGUL SYLLABLE NEUN
 726   0.000%  99.988%  U+0B9CC ‹만›  GC=Lo    HANGUL SYLLABLE MAN
 727   0.000%  99.988%  U+0C11C ‹서›  GC=Lo    HANGUL SYLLABLE SEO
 728   0.000%  99.989%  U+0C2B5 ‹습›  GC=Lo    HANGUL SYLLABLE SEUB
 729   0.000%  99.989%  U+0C601 ‹영›  GC=Lo    HANGUL SYLLABLE YEONG
 741   0.000%  99.989%  U+00441 ‹с›  GC=Ll    CYRILLIC SMALL LETTER ES
 742   0.000%  99.989%  U+00444 ‹ф›  GC=Ll    CYRILLIC SMALL LETTER EF
 743   0.000%  99.989%  U+004B0 ‹Ұ›  GC=Lu    CYRILLIC CAPITAL LETTER STRAIGHT U WITH STROKE

Of those 239 distinct trans-ASCII code points, 59 of them are also outside Unicode’s Basic Multilingual Plane, so any processing must be able to handle the full range of Unicode. All but one of these are mathematical letters. These are the top 20 of those:

rank  percent cumulative  code glyph  GC=??   Name
---------------------------------------------------------------------
 227   0.004%  99.660%  U+1D49E ‹𝒞›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL C
 240   0.003%  99.704%  U+1D4AF ‹𝒯›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL T
 252   0.003%  99.738%  U+1D4AE ‹𝒮›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL S
 275   0.002%  99.791%  U+1D49F ‹𝒟›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL D
 279   0.002%  99.799%  U+1D4B3 ‹𝒳›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL X
 289   0.002%  99.818%  U+1D4A9 ‹𝒩›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL N
 291   0.002%  99.821%  U+1D4AB ‹𝒫›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL P
 292   0.002%  99.823%  U+1D4A2 ‹𝒢›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL G
 313   0.001%  99.854%  U+1D49C ‹𝒜›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL A
 316   0.001%  99.858%  U+1D53C ‹𝔼›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL E
 341   0.001%  99.884%  U+1D4AA ‹𝒪›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL O
 430   0.000%  99.941%  U+1D4A5 ‹𝒥›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL J
 450   0.000%  99.948%  U+1D4A6 ‹𝒦›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL K
 458   0.000%  99.950%  U+1D4B1 ‹𝒱›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL V
 461   0.000%  99.951%  U+1D4B2 ‹𝒲›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL W
 468   0.000%  99.953%  U+1D4B4 ‹𝒴›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Y
 469   0.000%  99.954%  U+1D4B5 ‹𝒵›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Z
 500   0.000%  99.962%  U+1D4B0 ‹𝒰›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL U
 518   0.000%  99.966%  U+1D4AC ‹𝒬›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Q
 560   0.000%  99.973%  U+1D54A ‹𝕊›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL S

Other corpora will vary. You have to know your dataset.

3
  • Thanks for the comprehensive answer. I am aware of a way to find details about a unicode code point in python, using the unicodedata library. Is there a way to find out this information in JS?
    – varrunr
    May 13, 2012 at 22:00
  • @varrunr If you mean in Javascript, then you will have to use the XRegexp plugin to access the general categories.
    – tchrist
    May 13, 2012 at 22:45
  • If you're using server-side javascript with node.js, you can npm install unicode : github.com/dodo/node-unicodetable Jan 23, 2013 at 19:08
3

Unfortunately, this is close to impossible. It is very difficult to define which characters each language uses. (For example, English surely uses many characters outside \u0000 to \u007F, such as dashes and “é” in many words of French origin. Where do you draw the limits.) There are some character collections defined for languages in the CLDR database, but the choices there can be questioned. For many languages, the collections are so large and sparse (in terms of Unicode coding space) that any regular expressions for them would be quite long.

So hard-coding ranges would not even be enough; you would need sets of ranges plus individual characters.

Perhaps the most important question is: What would you use this for? The techniques need to be evaluated according to that. And in general, JavaScript is very primitive and limited when it comes to internationalization.

3
  • Restricting the range to the alphabet of the language is also good enough.
    – varrunr
    May 10, 2012 at 4:19
  • 1
    @varrunr, that change in the requirements partly makes the problem a little less impossible, partly makes it more complicated. The question “what is an alpabetic character?” is far from trivial. And do you mean that all non-alphabetic characters should be excluded? Even the space? May 10, 2012 at 4:29
  • @JukkaK.Korpela I’ve run some corpus analytics on the PubMed Open Access collection, a very large all-English dataset. The top five non-ASCII code points are EN DASH, NO-BREAK SPACE, PLUS-MINUS SIGN, MINUS SIGN, and EM SPACE. These are extremely frequent.
    – tchrist
    May 13, 2012 at 14:42
1

There's a powerful Unicode support in regular expressions: http://www.regular-expressions.info/unicode.html

But these features are supported in JavaScript only since es6. Which is not implemented even in Chrome yet. Maybe, it will be implemented when you finish your code.

Also, things are not so simple even for English language: café, naïve, coördinator.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.