171

I got strange error message when tried to save first_name, last_name to Django's auth_user model.

Failed examples

user = User.object.create_user(username, email, password)
user.first_name = u'Rytis'
user.last_name = u'Slatkevičius'
user.save()
>>> Incorrect string value: '\xC4\x8Dius' for column 'last_name' at row 104

user.first_name = u'Валерий'
user.last_name = u'Богданов'
user.save()
>>> Incorrect string value: '\xD0\x92\xD0\xB0\xD0\xBB...' for column 'first_name' at row 104

user.first_name = u'Krzysztof'
user.last_name = u'Szukiełojć'
user.save()
>>> Incorrect string value: '\xC5\x82oj\xC4\x87' for column 'last_name' at row 104

Succeed examples

user.first_name = u'Marcin'
user.last_name = u'Król'
user.save()
>>> SUCCEED

MySQL settings

mysql> show variables like 'char%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       | 
| character_set_connection | utf8                       | 
| character_set_database   | utf8                       | 
| character_set_filesystem | binary                     | 
| character_set_results    | utf8                       | 
| character_set_server     | utf8                       | 
| character_set_system     | utf8                       | 
| character_sets_dir       | /usr/share/mysql/charsets/ | 
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

Table charset and collation

Table auth_user has utf-8 charset with utf8_general_ci collation.

Results of UPDATE command

It didn't raise any error when updating above values to auth_user table by using UPDATE command.

mysql> update auth_user set last_name='Slatkevičiusa' where id=1;
Query OK, 1 row affected, 1 warning (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select last_name from auth_user where id=100;
+---------------+
| last_name     |
+---------------+
| Slatkevi?iusa | 
+---------------+
1 row in set (0.00 sec)

PostgreSQL

The failed values listed above can be updated into PostgreSQL table when I switched the database backend in Django. It's strange.

mysql> SHOW CHARACTER SET;
+----------+-----------------------------+---------------------+--------+
| Charset  | Description                 | Default collation   | Maxlen |
+----------+-----------------------------+---------------------+--------+
...
| utf8     | UTF-8 Unicode               | utf8_general_ci     |      3 | 
...

But from http://www.postgresql.org/docs/8.1/interactive/multibyte.html, I found the following:

Name Bytes/Char
UTF8 1-4

Is it means unicode char has maxlen of 4 bytes in PostgreSQL but 3 bytes in MySQL which caused above error?

1

9 Answers 9

170

None of these answers solved the problem for me. The root cause being:

You cannot store 4-byte characters in MySQL with the utf-8 character set.

MySQL has a 3 byte limit on utf-8 characters (yes, it's wack, nicely summed up by a Django developer here)

To solve this you need to:

  1. Change your MySQL database, table and columns to use the utf8mb4 character set (only available from MySQL 5.5 onwards)
  2. Specify the charset in your Django settings file as below:

settings.py

DATABASES = {
    'default': {
        'ENGINE':'django.db.backends.mysql',
        ...
        'OPTIONS': {'charset': 'utf8mb4'},
    }
}

Note: When recreating your database you may run into the 'Specified key was too long' issue.

The most likely cause is a CharField which has a max_length of 255 and some kind of index on it (e.g. unique). Because utf8mb4 uses 33% more space than utf-8 you'll need to make these fields 33% smaller.

In this case, change the max_length from 255 to 191.

Alternatively you can edit your MySQL configuration to remove this restriction but not without some django hackery

UPDATE: I just ran into this issue again and ended up switching to PostgreSQL because I was unable to reduce my VARCHAR to 191 characters.

7
  • 13
    this answer needs way, way, way more upvotes. Thanks! The real problem is your application may run fine for years until someone tries to enter a 4byte character. Apr 30, 2014 at 3:44
  • 4
    This is absolutely the right answer. The OPTIONS setting is critical to make django decode emoji characters and store them in MySQL. Just changing mysql charset to utf8mb4 via SQL commands is not enough!
    – Xerion
    Aug 6, 2015 at 4:24
  • 1
    There's no need to update the character set of the whole table to utf8mb4. Just update character set of necessary columns. Also the 'charset': 'utf8mb4' option in Django settings is critical, as @Xerion said. Finally, the index problem is a mess. Remove the index on the column, or make its length no more than 191, or use a TextField instead!
    – Rockallite
    Sep 23, 2016 at 9:53
  • 2
    I love your link to this quote: This is just another case of MySQL being purposefully and irreversibly brain-damaged. :)
    – Qback
    Jan 8, 2020 at 12:22
  • Note that if using mysql.connector.django as the database backend, you must also set 'collation': 'utf8mb4_unicode_ci' in OPTIONS.
    – alstr
    Aug 26, 2020 at 16:06
126

I had the same problem and resolved it by changing the character set of the column. Even though your database has a default character set of utf-8 I think it's possible for database columns to have a different character set in MySQL. Here's the SQL QUERY I used:

    ALTER TABLE database.table MODIFY COLUMN col VARCHAR(255)
    CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL;
3
  • 14
    Ugh, I changed all the character sets on everything I could until I really re-read this answer: columns can have their own character sets, independent of the tables and the database. That's crazy and also was exactly my problem.
    – markpasc
    Jul 18, 2011 at 17:43
  • 1
    This worked for me as well, using mysql with the defaults, in a TextField model.
    – madprops
    Sep 9, 2011 at 23:24
  • This solved my problem. The only change I did was to use utf8mb4 and utf8mb4_general_ci instead of utf8 / utf8_general_ci. Sep 28, 2019 at 14:06
72

If you have this problem here's a python script to change all the columns of your mysql database automatically.

#! /usr/bin/env python
import MySQLdb

host = "localhost"
passwd = "passwd"
user = "youruser"
dbname = "yourdbname"

db = MySQLdb.connect(host=host, user=user, passwd=passwd, db=dbname)
cursor = db.cursor()

cursor.execute("ALTER DATABASE `%s` CHARACTER SET 'utf8' COLLATE 'utf8_unicode_ci'" % dbname)

sql = "SELECT DISTINCT(table_name) FROM information_schema.columns WHERE table_schema = '%s'" % dbname
cursor.execute(sql)

results = cursor.fetchall()
for row in results:
  sql = "ALTER TABLE `%s` convert to character set DEFAULT COLLATE DEFAULT" % (row[0])
  cursor.execute(sql)
db.close()
8
  • 4
    This solution solved all my issues with a django app which was storing file and directory paths. Toss in dbname as your django database and let it run. Worked like a charm!
    – Chris
    Aug 5, 2012 at 22:39
  • 1
    This code didn't work for me until I added db.commit() before db.close(). Feb 28, 2013 at 22:29
  • 1
    Does this solution avoid the issue discussed in @markpasc comment: '...4-byte UTF-8 characters such as emoji in MySQL 5.1's 3-byte utf8 character set'
    – CatShoes
    Apr 24, 2013 at 16:51
  • the solution help me when i was deleting a record trough django admin, i didn't had any problem when creating o editing...weird! I was even able to delete directly in the db Oct 22, 2013 at 23:50
  • Should I do this every time I change the Model?
    – Vanuan
    May 21, 2014 at 18:25
28

If it's a new project, I'd just drop the database, and create a new one with a proper charset:

CREATE DATABASE <dbname> CHARACTER SET utf8;
5
  • Hi kindly help check this question stackoverflow.com/questions/46348817/…
    – King
    Oct 7, 2017 at 13:21
  • 1
    In my case, our db is created by docker so to fix I added the following to the db:command: instruction in my compose file: - --character-set-server=utf8
    – followben
    Apr 11, 2018 at 7:03
  • 1
    As simple as that. Thanks @Vanuan
    – Enkum
    Jul 18, 2018 at 16:05
  • if this is not a new project, we get backup from db, drop it and recreate it with utf8 charset and then restore backup. I did it in my project that was not new... Dec 3, 2019 at 15:11
  • @followben thank you, that resolved my problem. Though I had to use --character-set-server=utf8mb4
    – bilbohhh
    Jan 22, 2021 at 13:55
8

I just figured out one method to avoid above errors.

Save to database

user.first_name = u'Rytis'.encode('unicode_escape')
user.last_name = u'Slatkevičius'.encode('unicode_escape')
user.save()
>>> SUCCEED

print user.last_name
>>> Slatkevi\u010dius
print user.last_name.decode('unicode_escape')
>>> Slatkevičius

Is this the only method to save strings like that into a MySQL table and decode it before rendering to templates for display?

6
  • 12
    I'm having a similar problem, but I don't agree that this is a valid solution. When you .encode('unicode_escape') you're not actually storing unicode characters in the database. You're forcing all the clients to unencode before using them, which means it won't work properly with django.admin or all sorts of other things.
    – muudscope
    Apr 26, 2010 at 16:43
  • 3
    While it seems distasteful to store escape codes instead of characters, this is probably one of the few ways to save 4-byte UTF-8 characters such as emoji in MySQL 5.1's 3-byte utf8 character set.
    – markpasc
    Jun 27, 2012 at 15:22
  • 2
    There is an encoding called utf8mb4 that allows more than the Basic Multilingual Plane to be stored. I know, you'd think "UTF8" is all that's needed to store Unicode fully. Well, whaddaya know, it's not. See dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html Aug 21, 2013 at 21:03
  • @jack you might want to consider changing the accepted answer to one which is more useful
    – donturner
    Dec 3, 2013 at 11:13
  • it is a feasible workaround, but I do not recommend using it too (as advocated by @muudscope). I still can't store, for example, emoji to mysql databases. Has anybody accomplished it? Mar 17, 2014 at 9:33
6

You can change the collation of your text field to UTF8_general_ci and the problem will be solved.

Notice, this cannot be done in Django.

2

Improvement to @madprops answer - solution as a django management command:

import MySQLdb
from django.conf import settings

from django.core.management.base import BaseCommand


class Command(BaseCommand):

    def handle(self, *args, **options):
        host = settings.DATABASES['default']['HOST']
        password = settings.DATABASES['default']['PASSWORD']
        user = settings.DATABASES['default']['USER']
        dbname = settings.DATABASES['default']['NAME']

        db = MySQLdb.connect(host=host, user=user, passwd=password, db=dbname)
        cursor = db.cursor()

        cursor.execute("ALTER DATABASE `%s` CHARACTER SET 'utf8' COLLATE 'utf8_unicode_ci'" % dbname)

        sql = "SELECT DISTINCT(table_name) FROM information_schema.columns WHERE table_schema = '%s'" % dbname
        cursor.execute(sql)

        results = cursor.fetchall()
        for row in results:
            print(f'Changing table "{row[0]}"...')
            sql = "ALTER TABLE `%s` convert to character set DEFAULT COLLATE DEFAULT" % (row[0])
            cursor.execute(sql)
        db.close()

Hope this helps anybody but me :)

1
  • This line sql = "ALTER TABLE `%s` convert to character set DEFAULT COLLATE DEFAULT" % (row[0]) should be changed to sql = "ALTER TABLE `%s` convert to character set 'utf8' COLLATE 'utf8_unicode_ci' " % (row[0]) . Thank you for very best answer.
    – huyxdong
    Feb 23, 2022 at 4:56
1

You aren't trying to save unicode strings, you're trying to save bytestrings in the UTF-8 encoding. Make them actual unicode string literals:

user.last_name = u'Slatkevičius'

or (when you don't have string literals) decode them using the utf-8 encoding:

user.last_name = lastname.decode('utf-8')
1
  • @Thomas, i tried exactly as what you said but it still raise same errors.
    – jack
    Jan 21, 2010 at 11:54
0

Simply alter your table, no need to any thing. just run this query on database. ALTER TABLE table_nameCONVERT TO CHARACTER SET utf8

it will definately work.

1
  • Not straightforward: Error Code: 1118. Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs . I can change it, but then Django will mess with it May 13, 2021 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.