0

I'm trying to use Ruby Regexp on Tamil Unicode codepoints. Both \u0BC0 and \u0BCD are combining vowel markers in the Mark, Nonspacing [Mn] character category, which per my understanding should match the [:alpha:] class. But \u0BCD does not seem to match the class.

irb(main):002:0> "\u0BAE\u0BC0\u0BA9\u0BCD\u0BA9".scan(/[[:alpha:]]+/).each { |s| puts s.dump }
"\u0BAE\u0BC0\u0BA9"
"\u0BA9"
=> ["மீன", "ன"]
irb(main):003:0> 

I'm on Ruby 2.6.5p114 on OS X 10.15.2. What could be going on?

12
  • 2
    The simplest and most obvious explanation would be a bug in Regexp/Onigmo. Jan 12, 2020 at 21:12
  • Don't you want /[[:alpha:]]/? Jan 12, 2020 at 21:45
  • 1
    Searching for "posix :alpha:" shows lots of documentation that [:alpha:] is equivalent to [a-zA-Z] only. What happens if you use the Unicode equivalent [\p{L}\p{Nl}]? Jan 12, 2020 at 22:01
  • 1
    @theTinMan: The Ruby documentation for Regexp does not explicitly spell out what [[:alpha:]] matches, but it does say that the POSIX bracket expressions match non-ASCII characters, and it gives [[:digit:]] as an example, saying it matches anything with the Unicode property Nd. The Onigmo documentation explicitly specifies that [[:alpha:]] matches Letter or Mark. Jan 12, 2020 at 22:49
  • 1
    I found a difference between the two: unicode.org/cldr/utility/character.jsp?a=0BC0 says "alnum: Yes, Alphabetic: Yes", whereas unicode.org/cldr/utility/character.jsp?a=0BCD says "alnum: No, Alphabetic: No". So, it looks like [[:alpha:]] might be using one of those properties, either exclusively, or in conjunction with Letter and Mark. Jan 12, 2020 at 23:06

1 Answer 1

3

The two characters in question are (I have marked some interesting things in bold):

The Ruby documentation for the Regexp class does not explicitly spell out what [[:alpha:]] matches, but it does say that the POSIX bracket expressions match non-ASCII characters, and it gives [[:digit:]] as an example, saying it matches anything with the Unicode property Nd (Decimal Number).

While not explicitly documented, it makes sense to equate the Regexp POSIX bracket expression [[:alpha:]] with the Unicode property Alphabetic, which would mean that U+0BC0 matches and U+0BCD doesn't.

On the other hand, the documentation for Onigmo (the Regexp engine used in YARV, and mirrored in all other implementations) does explicitly specify the workings of [[:alpha:]]. In fact, it specifies it in two different places, and they contradict each other:

So, what seems to be going on, is that the Unicode Consortium does not consider U+0BCD to be alphabetic, and therefore, Onigmo and Ruby do not classify it as [[:alpha:]]. In that case, the Onigmo documentation is incorrect, and the Ruby documentation is imprecise.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.