1

I am facing an issue with the encode_json function in use JSON::XS qw(encode_json);. we have a web based project with Perl, JavaScript, jQuery and MySQL as the back end.

In a particular screen in the application, the name of the user is distorted.

The problem is the user name is in spanish and has accented characters. The table where we are saving the DB is old table and has charset latin 1, but when I am fetching the data, i am using cast and convert and encoding it to UTF-8 format.

select CONVERT(CAST(CONVERT(u.last_name USING  latin1) AS BINARY) USING utf8) AS last_name from user where user_id = 'XXX'

In the structure where I am storing the DB results, the data is appearing fine.

{ last_name => Cvas García}

this structure is passed to javascript function which renders the page

$data_json = encode_json( $data )

When I print $data_json, the accented characters are distorted once again.

"user_permissions":[{"last_name":"Cvas García"}]},

Can anyone help me how can I solve this, either in the Mysql or perl or javascript code?

2 Answers 2

2

The charset of the table is irrelevant. MySQL will automatically convert from the table's encoding to the connection's encoding. Then, you'll have to decode it on the Perl side, either explicitly or by passing mysql_enable_utf8 => 1 to the connect. I recommend the latter.

use utf8;                             # Source code encoded using UTF-8.
use open ':std', ':encoding(UTF-8)';  # Terminal expects/provides UTF-8.

use DBI      qw( );
use JSON::XS qw( encode_json );

my $user_id = "...";

my $dsn = "...";
my $user = "...";
my $password = "...";

my $dbh = DBI->connect("dbi:mysql:".$dsn, $user, $password, {
   PrintError => 0,
   RaiseError => 1,
   mysql_enable_utf8 => 1,  # Switch to UTF-8 for communication and decode.
});

my $name = $dbh->selectrow_array("
   SELECT `last_name`
     FROM `user`
    WHERE `user_id` = ?
", undef, $user_id);

# Encoding being performed by "use open".
print(sprintf("%vX", $name), "\n");        # 47.61.72.63.ED.61
print($name, "\n");                        # García

# Encoding being performed by "use open".
my $json = JSON::XS->new->encode([$name]);
print(sprintf("%vX", $json), "\n");        # 5B.22.47.61.72.63.ED.61.22.5D
print($json, "\n");                        # ["García"]

binmode(STDOUT); # Override "use open".
my $json_utf8 = JSON::XS->new->utf8->encode([$name]);  # aka encode_json([$name])
print(sprintf("%vX", $json_utf8), "\n");   # 5B.22.47.61.72.63.C3.AD.61.22.5D
print($json_utf8, "\n");                   # ["García"]
3
  • Thanks for the suggestion. I am not getting the same output as mentioned in your code. Here is what I am getting as output. 43.75.65.76.61.73.20.47.61.72.63.ED.61 Cuevas García 5B.22.43.75.65.76.61.73.20.47.61.72.63.ED.61.22.5D ["Cuevas García"] 5B.22.43.75.65.76.61.73.20.47.61.72.63.C3.AD.61.22.5D ["Cuevas García"]... Is the initial data saved into user table is itself corrupt? Nov 9, 2015 at 14:17
  • The hex show the strings are correct, so you simply didn't specify the correct encoding for your terminal.
    – ikegami
    Nov 9, 2015 at 14:24
  • ED is the latin1 encoding; C3 AD is the utf8 encoding.
    – Rick James
    Dec 7, 2015 at 22:15
-1

Try this

use JSON;
use utf8;
use Data::Printer;
utf8::encode($data) unless (utf8::is_utf8($data));
p encode_json($data);
0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.