1

I just learned about character sets today, so forgive the newb factor if this is confusing. Please ask for clarification if it's needed.

I wrote a program in php which recursively goes through the files in a folder and stores the file names in a database. The file names are then all exported from the database in json format using the json_encode($array) function.

However this function only works with UTF-8 encoded data. And since a few of the key-value pairs in the json export have the value of null, I'm lead to believe that those strings of filenames taken from the database are in fact not utf-8.

I've ensured that all the data going in and out of the the database is utf-8 by setting the defaults to utf-8 in my.cnf and restarting mysql from the command line using service mysql restart

[client]
default-character-set=utf8

[mysqld]
default-character-set = utf8

I then created my database, the table and all the columns in the table and confirmed that the database, table and all the columns are in fact utf-8

Checks if database is utf-8

SELECT default_character_set_name FROM information_schema.SCHEMATA S
WHERE schema_name = "schemaname";

Checks if table is utf-8

SELECT CCSA.character_set_name FROM information_schema.`TABLES` T,
       information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
WHERE CCSA.collation_name = T.table_collation
  AND T.table_schema = "schemaname"
  AND T.table_name = "tablename";

Checks if field is utf-8

SELECT character_set_name FROM information_schema.`COLUMNS` C
WHERE table_schema = "schemaname"
  AND table_name = "tablename"
  AND column_name = "columnname";

There's this file that has the characters –µ–ª–∫—É–Ω—á–∏–∫ in the file name. When it's stored in the database the values appear as –©–µ–ª–â'.

Per my database settings, are all the strings going in and out of my database utf-8?

What can I do to ensure the data I am SELECT'ing from the database is utf-8, so I can perform json_encode($array)? (NOTE: this function only works on utf-8 encoded data)

1 Answer 1

1

Unfortunately I don't know how you can ensure everything coming out is UTF-8 (now I'm curious too!), but a starting point would be trying this in your PHP:

$encodedNames = array();
$errors = array();

// Loop through all of the filenames
foreach($filenames as $filename)
{
  // Check if it's UTF-8 encoded
  if('UTF-8' === mb_detect_encoding($filename, 'UTF-8', true))
  {
    $encodedNames[] = $filename;
  }
  else
  {
    $errors[] = $filename;
  }
}

// json_encode the UTF-8 filenames
$jsonString = json_encode($encodedNames);

// Log the other filenames here so you can deal with them later...

http://php.net/manual/en/function.mb-detect-encoding.php

3
  • Thanks Steve, will try this out soon. As far as character encodings are concerned, doesn't the proper database setup ensure that the strings stored in the database are utf-8? Is it just when they're stored in a php array that they become non-utf8?
    – user784637
    Feb 19, 2012 at 2:51
  • I'm not really sure; I would think that you're right in that a proper database setup ensures proper encoding (I'm not really a DB guy), but I also just saw mysql_set_charset - maybe it has something to do with that? php.net/manual/en/function.mysql-set-charset.php
    – Steve
    Feb 19, 2012 at 2:59
  • Hey Steve, here's something interesting I just learned - MySQL PDO ignores the charset specification in it's dsn, so you have to manualy set the name to utf-8 upon connecting.
    – user784637
    Feb 19, 2012 at 4:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.