0

I have an app that uses Cyrillic (Macedonian) alphabet. I have an alphabet menu with all of the letters (manually typed as an array which I'm calling it from a helper), that links to items where the first character is that letter. So, it seems that "К" and "Ќ" (also "Г" and "Ѓ") are listing the same items, as if they were using the same letter. Works great in development, not sure why it does this on production. I have set UTF-8 encoding on production database.

Here's the prod log. It's not getting the same character.

App 18197 stderr: Started GET "/letterfilter?title=%D0%8C" for IP at2015-07-30 12:03:46 -0400
App 18197 stderr: Processing by PostsController#letterfilter as HTML
App 18197 stderr:   Parameters: {"title"=>"Ќ"}
App 18197 stderr:   Rendered posts/letterfilter.html.haml within layouts/application (4.3ms)
App 18197 stderr:   Rendered posts/_search.html.haml (0.8ms)
App 18197 stderr:   Rendered shared/_header.html.haml (9.6ms)
App 18197 stderr:   Rendered shared/_footer.html.haml (0.2ms)
App 18197 stderr: Completed 200 OK in 18ms (Views: 16.6ms | ActiveRecord: 0.2ms)

App 18197 stderr: Started GET "/letterfilter?title=%D0%9A" for IP at 2015-07-30 12:03:51 -0400
App 18197 stderr: Processing by PostsController#letterfilter as HTML
App 18197 stderr:   Parameters: {"title"=>"К"}
App 18197 stderr:   Rendered posts/letterfilter.html.haml within layouts/application (4.9ms)
App 18197 stderr:   Rendered posts/_search.html.haml (0.7ms)
App 18197 stderr:   Rendered shared/_header.html.haml (7.7ms)
App 18197 stderr:   Rendered shared/_footer.html.haml (0.2ms)
App 18197 stderr: Completed 200 OK in 17ms (Views: 14.2ms | ActiveRecord: 0.9ms)

What might be causing this issue? Should I update my database encoding to utf8mb4?

Any help is welcomed. Thanks.

2
  • If the problem at mysql or at rails end? Jul 31, 2015 at 9:42
  • @ManishShrivastava I'm not sure, really. It works as intended on local server, but it different in production env. Jul 31, 2015 at 10:30

1 Answer 1

0

Ќ is hex D08C in utf8 or utf8mb4. Cyrillic is completely covered by either CHARACTER SET. К is D09A, as can be seen in the ?title=

Hmmm, this is quite interesting:

mysql> SELECT 'К' = 'Ќ' COLLATE utf8_bin AS bin,
              'К' = 'Ќ' COLLATE utf8_general_ci AS general,
              'К' = 'Ќ' COLLATE utf8_unicode_ci AS unicode;
+-----+---------+---------+
| bin | general | unicode |
+-----+---------+---------+
|   0 |       1 |       0 |
+-----+---------+---------+

mysql> SELECT 'Г' = 'Ѓ' COLLATE utf8_bin AS bin,
              'Г' = 'Ѓ' COLLATE utf8_general_ci AS general,
              'Г' = 'Ѓ' COLLATE utf8_unicode_ci AS unicode;
+-----+---------+---------+
| bin | general | unicode |
+-----+---------+---------+
|   0 |       1 |       0 |
+-----+---------+---------+

The bit patterns are different, so utf8_bin collates unequal. But usually whenever general is equal, so is unicode.

Back to your question... What do you mean "it's not getting the same character"?

6
  • Thanks. By not getting the same character I mean it's not rendering only "Ќ" items when I select "Ќ", but "К" as well. I kinda messed up my explanation there, sorry if it was confusing. I'll try to solve this one more time, or I'll just update to Postgre hoping it won't have the same issue. :) Aug 1, 2015 at 8:49
  • utf8 (or utf8mb4) needs to be specified in several places, inside and outside MySQL. Is Ќ incorrectly displaying as К (Cyrillic KA, not Latin K)? I don't think SELECT can every make such a change; I suspect something in the display mechanism.
    – Rick James
    Aug 1, 2015 at 14:58
  • Both К and Ќ are displaying the same results, items that start on those letters. Here's the query, the show is the usual setup. def letterfilter @letter = Post.where(["title like ? AND approved IS TRUE", "#{params[:title]}%"]) end edit: format Aug 1, 2015 at 15:03
  • And if it helps, here's the method from the alphabet helper. def alphabet cyrillic_alphabet = ["А", "Б", "В", "Г", "Д", "Ѓ", "Е", "Ж", "З", "Ѕ", "И", "Ј", "К", "Л", "Њ", "М", "Н", "Љ", "О", "П", "Р", "С","Т", "Ќ", "У", "Ф", "Х", "Ц", "Ч", "Џ", "Ш"] end Aug 1, 2015 at 15:05
  • You are leaving out D086 1030=x0406 [І] L CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I D087 1031=x0407 [Ї] L CYRILLIC CAPITAL LETTER YI D08B 1035=x040B [Ћ] L CYRILLIC CAPITAL LETTER TSHE D08E 1038=x040E [Ў] L CYRILLIC CAPITAL LETTER SHORT U (and maybe others)? (I am not familiar with the details of the Cyrillic alphabet.)
    – Rick James
    Aug 1, 2015 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.