0

I am migrating a MySQL database that was created in LATIN1 to UTF8. To do that, I am first changing each column to the corresponding binary type and then to UTF8:

ALTER TABLE clientes CHARACTER SET utf8;
ALTER TABLE clientes change nombre nombre varbinary(255);
ALTER TABLE clientes change nombre nombre varchar(255) character set utf8;

Since, according to all docs, that's the right way to prevent data from being mangled...

...However, the data still gets mangled. I'll just put two examples:

  • The word "Larrasoaña" gets truncated at the "ñ", with the error: Warning: #1366 Incorrect string value: '\xF1a' for column 'nombre'
  • The word "Jesús y María" gets truncated at the "ú", with the error Warning: #1366 Incorrect string value: '\xFAs y M...' for column 'nombre'

How did that data get in? Well, the DB is the backend to a PHP web app, which used UTF8 for everything (including connecting to the MySQL server with "SET NAMES UTF8")... except creating the database properly. So I assume all the data that was added in was in UTF8.

To summarize: it seems that I had UTF8 text stored in LATIN1 columns, and now that I try to change the columns to UTF8, the text gets truncated.

Why is this happening? What can I do?

EDIT: forgot to mention, I'm doing all of this from PhpMyAdmin, since I don't have command line access.

1 Answer 1

1

F1 and FA are latin1 encodings. You need to tell MySQL that the data is latin1. One way is via SET NAMES latin1.

But note... That is independent of the setting for the column you are trying to store the data into. And, these days, utf8mb4 is the preferred setting for text. MySQL will convert between the column's encoding and the client's encoding. But you must tell it the client's encoding via connection parameters (or SET NAMES).

The pair of ALTER TABLEs works for certain situations, not all situations! You probably wanted the first entry in http://mysql.rjweb.org/doc.php/charcoll#fixes_for_various_cases

Table is CHARACTER SET latin1 and correctly encoded in latin1; want utf8mb4:

ALTER TABLE tbl CONVERT TO CHARACTER SET utf8mb4;

I don't happen to know if your data is irreparably hosed. Please provide one of the lines, together with HEX.

Hex

"Larrasoaña" is encoded as 4C61727261736F61F161, and "Jesús y María" as 4A6573FA732079204D6172ED6120

Those are latin1-encoded (or latin5 or dec8). If the table definition (SHOW CREATE TABLE) says latin1, then you could leave it alone. (latin1 handles Western European languages, but not Asian.)

If you want to convert all the text columns to utf8 or utf8mb4, do an ALTER like the one I presented above. Your 3-Alter approach will not work correctly; it assumes the bytes in the latin1 column are really UTF-8 bytes (which they aren't).

But... You must specify the client's encoding based on what the client wants. And it does not matter whether the client and the table agree since conversion will be provided.

Why the 3-step Alter fails

ALTER TABLE clientes CHARACTER SET utf8; -- This sets the default charset for new columns. It has no effect on the existing column definitions and any data in those columns.

ALTER TABLE clientes change nombre nombre varbinary(255); -- This says "forget about any text encoding". That is F1 is now just a bunch of bits, not the latin1 representation for ñ.

ALTER TABLE clientes change nombre nombre varchar(255) character set utf8; -- This takes those varbinary bits and says "let's treat them as utf8. And that gives the error message because F1 is not a valid encoding for utf8.

That procedure is appropriate if the bytes are already utf8 bytes. That is, if it were already the 2-byte C3B1 for ñ. (By the way, this usually manifests itself as 'Mojibake', displaying as ñ when interpreted as latin1.)

The 1-Alter procedure...

ALTER TABLE clientes CONVERT TO CHARACTER SET utf8; (to convert the entire table) or ALTER TABLE clientes MODIFY nombre varchar(255) character set utf8; (to convert just one column). They do the following things:

For each text (char/varchar/text) column, it reads the data according to its current encoding (latin1, F1), converts it to utf8 (or utf8mb4) (C3B1) and writes back into the row. Meanwhile, it has changed the declaration to be CHARACTER SET utf8.

That is, it is the 'right' process for changing the CHARACTER SET without changing the "text". True, the encoding changed (F1 -> C3B1), but that is in keeping with the change to the CHARACTER SET.

Recovery

Your first 2 ALTERs worked, correct? Did the 3rd one succeed, fail, or leave a messed up table?

If it aborted, leaving varbinary in place, then do 2 more alters: First go back to latin1; then go straight to utf8.

If it left you with a messed up column, especially if rows are truncated, then you need to go back to a backup, or otherwise reload the data.

12
  • Well, I had backups and it was a test copy of the DB (not the production site) ;-), so there wasn't anything "irrepairable". Anyway, I restored from backups and tried again, without converting to binary; I just put the "ALTER TABLE clientes change nombre nombre varchar(255) character set utf8;" command... and it seems to have worked: no errors, and the text seems to be okay. Is there anything else I should have done? I didn't use "ALTER TABLE ... CONVERT TO CHARACTER SET utf8", because I didn't know whether to execute it before or after changing the column type.
    – PaulJ
    Dec 19, 2019 at 19:03
  • As for using "SET NAMES", I'm using PhpMyAdmin on a hosted site (I have no control over the PhpMyAdmin installation), so it seems difficult. Is there a way to do so from a hosted PhpMyAdmin?
    – PaulJ
    Dec 19, 2019 at 19:05
  • @PaulJ - If you are using LOAD DATA, it has a CHARACTER SET clause. Or are you using some other feature of PhpMyAdmin?
    – Rick James
    Dec 19, 2019 at 20:19
  • I don't understand. The data is already in the DB; I don't need to use LOAD DATA. The way I restored the DB was to nuke the entire DB and then load the Mysqldump backup I had made using the "Import" option in PhpMyAdmin. This way I got the original DB, with the LATIN1 columns, and I modified them as mentioned in my comment above.
    – PaulJ
    Dec 19, 2019 at 20:45
  • 1
    @PaulJ - I added a bunch to my Answer.
    – Rick James
    Dec 20, 2019 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.