3

How do I make a diacritic insensitive,

ex this persian string with diacritics

هواى بَر آفتابِ بارِز

is not the same as with removed diacritics in mySql

هواى بر آفتاب بارز

Is there a way of telling mysql to ignore the diacritics or do I have to remove all the diacritics in my fields manually?

5 Answers 5

3

It's a bit like case-insensitivity problem.

SELECT * FROM blah WHERE UPPER(foo) = "THOMAS"

Just convert both strings to diacritic-free before comparing.

3
  • (Not sure how to remove diacritics in SQL, perhaps someone else can help?)
    – Artelius
    Oct 18, 2009 at 10:41
  • Actually it's a little more complex than case-sensitivity or insensitivity. Unless a software includes support for specific scripts (which MySQL doesn't seem to have for Persian) you're out of luck. Or you're in for some work writing a custom function to do it for you; won't be pretty either way.
    – Joey
    Oct 18, 2009 at 10:57
  • It works in MySQL and Hebrew: (SELECT * FROM dictionary WHERE upper(nekudot)="כדי";) finds כְּדֵי. Thanks!
    – xerostomus
    Dec 9, 2021 at 7:16
2

I'm using utf8 (utf8_general_ci) and searching arabic without diacritics doesn't work, it isn't insensitive or it is but don't work properly.

I tried looking at the character with and without a diacritic using Hex and it looks like mysql considering it as two distinct character.

I'm thinking about using hex and replace (a lot of replace) to search for words while filtering diacritics.

my solution to have an insensitive search for arabic words:

SELECT arabic_word FROM Word
WHERE
REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(HEX(REPLACE(
arabic_word, "-", "")), "D98E", ""), "D98B", ""), "D98F", ""), "D98C", 
""),"D991",""),"D992",""),"D990",""),"D98D","") LIKE ?', '%'.$search.'%'

the values formatted in hexadecimal are diacritics that we want to filter. ugly but I didn't find another anwser.

0

Did you already read all of MySQL Character Set Support to check if the answer to your question isn't already there? Especially collations are to be understood.

I wild guess is that using utf8_general_ci could do the right thing for you

1
0

Setting

set names 'utf8'

before making a query usually does the trick for latin lookups. I'm not sure if this works for arabic as well.

0

The cleanest solution I've come to is:

SELECT arabic_word 
FROM Word
WHERE ( arabic_word REGEXP '{$search}' OR SOUNDEX( arabic_word ) = SOUNDEX( '{$search}' ) );

I haven't checked the costs of the SOUNDEX function. I guess this could for small tables, but not for large datasets.

1
  • 1
    This will do better: SET NAMES utf8; SELECT arabic_word FROM Word WHERE arabic_word LIKE '{$search}'; I'm used to use REGEXP, but in this case LIKE is the key. Must read: stackoverflow.com/questions/3304464/…
    – Marcos
    Aug 22, 2012 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.