-1

I'm fairly confused about encodings in python. I have the following string.

s = "Caf\xe9/Coffee/Tea"

I want to make it a unicode string so that it will display properly. The following works:

t = u"Caf\xe9/Coffee/Tea"

print t

The output is "Café/Coffee/Tea"

However if instead I try

r = unicode(s)

I get the error "UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 3: ordinal not in range(128)"

I'm not even trying to display the unicode string to the console when I do this (what I recently learned is called a 'heisenbug'). But it seems my console can print unicode so I really don't understand the issue.

This is python 2.7 if that matters.

4
  • docs.python.org/2/howto/unicode.html try looking in here. Mar 30, 2015 at 4:27
  • that string (bytes) is already encoded. you need to .decode it to get a unicode object.
    – wim
    Mar 30, 2015 at 4:30
  • I'd recommend switching to Python 3 if nothing holds you back to Python 2 - you'd have much less confusion with Unicode there. Mar 30, 2015 at 5:49
  • yeah. i'd like to switch to python 3, but i'm using the anaconda distribution of python... and that's still on 2.7.
    – Ben
    Mar 31, 2015 at 3:28

4 Answers 4

3

Use

>>> "Caf\xe9/Coffee/Tea".decode('iso-8859-1')

Though I suggest that if possible use UTF-8 everywhere for Unicode encoding.

6
  • Hem, I do not know what is the charset you're using, it is not latin-1; \xf1 in Latin 1 is ñ Mar 30, 2015 at 4:30
  • I don't think 'iso-8859-1' is the right encoding. That will give you ñ instead of é.
    – wim
    Mar 30, 2015 at 4:32
  • é is \xe9, but ñ is \xf1
    – Aaron
    Mar 30, 2015 at 4:38
  • @Antti. I made a mistake pasting that in. Should have been "Caf\xe9/Coffee/Tea". Sorry about that.
    – Ben
    Mar 30, 2015 at 4:41
  • 1
    Yes, @Aaron, \xf1 is never é in any code page, just tested. Mar 30, 2015 at 4:41
1

Declare the encoding of your source file and you can write the Unicode characters directly in the file:

# coding: utf8
s = 'Café/Coffee/Tea'
print repr(s)
print s
t = u'Café/Coffee/Tea'
print t

Output (Note my console uses encoding cp437):

'Caf\xc3\xa9/Coffee/Tea'
Café/Coffee/Tea
Café/Coffee/Tea

The first line is a byte string. It will be in the source encoding. The second line does not print correctly since UTF-8 was sent to a code page 437 terminal. Python knows to encode the Unicode string to the console encoding, so the 3rd line prints correctly.

Here the source is declared and saved in a different encoding. Note printing Unicode is still correct:

# coding: iso-8859-1
s = 'Café/Coffee/Tea'
print repr(s)
print s
t = u'Café/Coffee/Tea'
print t

Output:

'Caf\xe9/Coffee/Tea'
CafΘ/Coffee/Tea
Café/Coffee/Tea
0

Decode that string using "Latin-1' , to Unicode.

>>> s = "Caf\xe9/Coffee/Tea"
>>> r = unicode(s, 'Latin-1')
>>> print r

Café/Coffee/Tea
0

unicode(bytestring) functions tries to decode the input bytestring using the default encoding on Python 2 ('ascii'). There are bytes outside ascii range and therefore it fails.

Note "\xf1" and u"\xf1" are very different on Python 2. The former is a bytestring, the latter is a Unicode string:

>>> "\xf1".decode('cp866')
u'\u0451'
>>> "\xf1".decode('cp437')                                                                                
u'\xb1'
>>> "\xf1".decode('latin-1')                                                                              
u'\xf1'
>>> u"\xf1".encode('cp866')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/encodings/cp866.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xf1' in position 0: character maps to <undefined>
>>> u"\xf1".encode('cp437')
'\xa4'
>>> u"\xf1".encode('latin-1')
'\xf1'

There Ain't No Such Thing as Plain Text. If you want to convert bytes into text; you should specify the character encoding.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.