I have a browser which sends utf-8 characters to my Python server, but when I retrieve it from the query string, the encoding that Python returns is ASCII. How can I convert the plain string to utf-8?

NOTE: The string passed from the web is already UTF-8 encoded, I just want to make Python to treat it as UTF-8 not ASCII.

share|improve this question
    
Try this link http://evanjones.ca/python-utf8.html – Mudassir Nov 15 '10 at 8:33
    
I think a better title would be How to coerce a string to unicode without translation? – boatcoder Aug 11 '16 at 22:05
up vote 177 down vote accepted
>>> plain_string = "Hi!"
>>> unicode_string = u"Hi!"
>>> type(plain_string), type(unicode_string)
(<type 'str'>, <type 'unicode'>)

^ This is the difference between a byte string (plain_string) and a unicode string.

>>> s = "Hello!"
>>> u = unicode(s, "utf-8")

^ Converting to unicode and specifying the encoding.

share|improve this answer
23  
,I am getting the following error: UnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 2: invalid start byte This is my code: ret=[] for line in csvReader: cline=[] for elm in line: unicodestr = unicode(elm, 'utf-8') cline.append(unicodestr) ret.append(cline) – Gopakumar N G Oct 22 '13 at 6:56
27  
None of this applies in Python 3, all strings are unicode and unicode() doesn't exist. – Noumenon Aug 28 '15 at 12:00
    
Kind of bumping this, but thanks. This fixed an issue where I was trying to print unicode and was getting �s. – 智障的人 Feb 7 '16 at 17:53
    
How to you convert u back to a str format (convert u back to s)? – Tanguy Aug 25 '17 at 13:25
    
@Tanguy thisIsAString = u'abcd'.encode('utf-8') – MoonKnight Oct 3 '17 at 15:46

If the methods above don't work, you can also tell Python to ignore portions of a string that it can't convert to utf-8:

stringnamehere.decode('utf-8', 'ignore')
share|improve this answer
4  
Uff, thanks alot, this peace of code finally ended my minidom trouble! – Alex Feb 19 '16 at 12:58
12  
peace of code .... not sure if typo... – user1717828 Apr 14 '17 at 13:00

Might be a bit overkill, but when I work with ascii and unicode in same files, repeating decode can be a pain, this is what I use:

def make_unicode(input):
    if type(input) != unicode:
        input =  input.decode('utf-8')
        return input
    else:
        return input
share|improve this answer

If I understand you correctly, you have a utf-8 encoded byte-string in your code.

Converting a byte-string to a unicode string is known as decoding (unicode -> byte-string is encoding).

You do that by using the unicode function or the decode method. Either:

unicodestr = unicode(bytestr, encoding)
unicodestr = unicode(bytestr, "utf-8")

Or:

unicodestr = bytestr.decode(encoding)
unicodestr = bytestr.decode("utf-8")
share|improve this answer

Adding the following line to the top of your .py file:

# -*- coding: utf-8 -*-

allows you to encode strings directly in your script, like this:

utfstr = "ボールト"
share|improve this answer
1  
It is not what OP asks. But avoid such string literals anyway. It creates Unicode string in Python 3 (good) but it is a bytestring in Python 2 (bad). Either add from __future__ import unicode_literals at the top or use u'' prefix. Don't use non-ascii characters in bytes literals. To get utf-8 bytes, you could utf8bytes = unicode_text.encode('utf-8') later if it is necessary. – jfs Apr 26 '15 at 1:26

In Python 3.6, they do not have a built-in unicode() function. To convert a string to unicode, simply get the unicode value of the character, and do this:

my_str = "\u221a25"
my_str = u"{}".format(my_str)
print(my_str)
>>> √25
share|improve this answer
city = 'Ribeir\xc3\xa3o Preto'
print city.decode('cp1252').encode('utf-8')
share|improve this answer
    
After hours of trying to read a file with city names, this finally worked. – Christian Nov 23 '17 at 9:40

Translate with ord() and unichar(). Every unicode char have a number asociated, something like an index. So Python have a few methods to translate between a char and his number. Downside is a ñ example. Hope it can help.

>>> C = 'ñ'
>>> U = C.decode('utf8')
>>> U
u'\xf1'
>>> ord(U)
241
>>> unichr(241)
u'\xf1'
>>> print unichr(241).encode('utf8')
ñ
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.