64

A previous LOAD DATA INFILE was run under the assumption that the CSV file is latin1-encoded. During this import the multibyte characters were interpreted as two single character and then encoded using utf-8 (again).

This double-encoding created anomalies like ñ instead of ñ.

How to correct these strings?

2
  • @Esailija It is not a MySQL function. It can be solved withoput bringing tools like PHP into the picture. (The question was created to be self-answered, but if a better solution comes up I will accept it instead of mine).
    – vbence
    Commented Jul 11, 2012 at 15:59
  • good to know, mark this as favorite so i can find it when i going to need it
    – Puggan Se
    Commented Jul 11, 2012 at 16:02

5 Answers 5

123

The following MySQL function will return the correct utf8 string after double-encoding:

CONVERT(CAST(CONVERT(field USING latin1) AS BINARY) USING utf8)

It can be used with an UPDATE statement to correct the fields:

UPDATE tablename SET
    field = CONVERT(CAST(CONVERT(field USING latin1) AS BINARY) USING utf8);
10
  • 1
    I got this working mostly, but found a sequence that does not work: letter ě is C49B but appears in my database as c384c29b and SELECT HEX(CONVERT(CAST(0xc384c29b AS CHAR) USING latin1)) got the invalid UTF-8 byte sequence C43F which means your outermost CONVERT does not work. UTF-8 bytes c29b should be Unicode 9B but MySQL is setting it to 3F (?) presumably because this is a control character in latin1. Perl's utf8::decode worked with it though.
    – hood
    Commented Nov 11, 2014 at 5:00
  • 8
    I can't refrain from saying how happy I am I've found this solution :) Commented Oct 5, 2015 at 14:51
  • Great. This seems to work. Now my challenge is to work out whether only some records have this encoding error, or all of them. I might need to only search for records with incorrect characters.
    – Simon E.
    Commented Jul 15, 2016 at 7:34
  • WoW ... i'm try to accomplish this via PHP but you saved my times. 100% worked in my database.
    – Abbas
    Commented Oct 14, 2016 at 12:10
  • Thank you! Everyone should see Eric’s enhancement which prevents data loss due to null results (which you could get if a specific column was already non-ASCII utf8, i.e. not latin1).
    – caw
    Commented Jan 16, 2019 at 3:52
19

The above answer worked for some of my data, but resulted in a lot of NULL columns after running. My thought is if the conversion wasn't successful it returns null. To avoid that, I added a small check.

UPDATE
    tbl

SET
    col =
    CASE
        WHEN CONVERT(CAST(CONVERT(col USING latin1) AS BINARY) USING utf8) IS NULL THEN col
        ELSE CONVERT(CAST(CONVERT(col USING latin1) AS BINARY) USING utf8)
    END
1
  • 3
    A really important addition to prevent data loss, thanks! You could simplify this further by replacing CASE WHEN converted IS NULL THEN original ELSE converted END with IF(converted IS NULL, original, converted) or even IFNULL(converted, original).
    – caw
    Commented Jan 16, 2019 at 3:58
5

well it is very important to use "utf8mb4" instead of "utf8" since mysql will strip out all the data after an unrecognized character. So the safer method is;

UPDATE tablename SET
field = CONVERT(CAST(CONVERT(field USING latin1) AS BINARY) USING utf8mb4);

be careful about this.

2
  • also if your collation is not utf8mb4_unicode_ci , first change this : ALTER TABLE tablename modify fieldname type CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
    – burkul
    Commented Nov 5, 2019 at 9:26
  • This is an old thread but I had exactly same problem and your answer saved my day! Thank you so much!
    – Sudhir
    Commented Jul 28, 2022 at 6:44
3

I meet this issue too, here a solution for Oracle:

update tablename t set t.colname = convert(t.colname, 'WE8ISO8859P1', 'UTF8') where t.colname like '%Ã%'

And another one for Java:

public static String fixDoubleEncoded(String text) {
    final Pattern pattern = Pattern.compile("^.*Ã[^0-9a-zA-Z\\ \t].*$");
    try {
        while (pattern.matcher(text).matches())
            text = new String(text.getBytes("iso-8859-1"), "utf-8");
    }
    catch (UnsupportedEncodingException e) {
        e.printStackTrace();
    }
    return text;
}
0

This is for generating queries for fixing it for a whole database with optional table prefix:

SELECT CONCAT('UPDATE `',table_name,'` SET `',column_name,'`=CONVERT(CAST(CONVERT(`',column_name,'` USING latin1) AS BINARY) USING utf8mb4);')
FROM information_schema.COLUMNS
WHERE table_schema="database_name" AND table_name LIKE "tableprefix%" AND data_type IN("varchar", "longtext", "mediumtext", "text", "char");

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.