66

Lately, I've had lots of trouble with __repr__(), format(), and encodings. Should the output of __repr__() be encoded or be a unicode string? Is there a best encoding for the result of __repr__() in Python? What I want to output does have non-ASCII characters.

I use Python 2.x, and want to write code that can easily be adapted to Python 3. The program thus uses

# -*- coding: utf-8 -*-
from __future__ import unicode_literals, print_function  # The 'Hello' literal represents a Unicode object

Here are some additional problems that have been bothering me, and I'm looking for a solution that solves them:

  1. Printing to an UTF-8 terminal should work (I have sys.stdout.encoding set to UTF-8, but it would be best if other cases worked too).
  2. Piping the output to a file (encoded in UTF-8) should work (in this case, sys.stdout.encoding is None).
  3. My code for many __repr__() functions currently has many return ….encode('utf-8'), and that's heavy. Is there anything robust and lighter?
  4. In some cases, I even have ugly beasts like return ('<{}>'.format(repr(x).decode('utf-8'))).encode('utf-8'), i.e., the representation of objects is decoded, put into a formatting string, and then re-encoded. I would like to avoid such convoluted transformations.

What would you recommend to do in order to write simple __repr__() functions that behave nicely with respect to these encoding questions?

0

3 Answers 3

42

In Python2, __repr__ (and __str__) must return a string object, not a unicode object. In Python3, the situation is reversed, __repr__ and __str__ must return unicode objects, not byte (née string) objects:

class Foo(object):
    def __repr__(self):
        return u'\N{WHITE SMILING FACE}' 

class Bar(object):
    def __repr__(self):
        return u'\N{WHITE SMILING FACE}'.encode('utf8')

repr(Bar())
# ☺
repr(Foo())
# UnicodeEncodeError: 'ascii' codec can't encode character u'\u263a' in position 0: ordinal not in range(128)

In Python2, you don't really have a choice. You have to pick an encoding for the return value of __repr__.

By the way, have you read the PrintFails wiki? It may not directly answer your other questions, but I did find it helpful in illuminating why certain errors occur.


When using from __future__ import unicode_literals,

'<{}>'.format(repr(x).decode('utf-8'))).encode('utf-8')

can be more simply written as

str('<{}>').format(repr(x))

assuming str encodes to utf-8 on your system.

Without from __future__ import unicode_literals, the expression can be written as:

'<{}>'.format(repr(x))
13
  • 2
    @EOL, The return value must be a string object. is how the reference manual page you point to expresses the constraint that the return value must be an instance of str (a unicode object would not be "a string object"). repr is normally expected to return ascii only (thing of repr(uo) for all unicode objects, for example: even that returns ascii only -- I think no built-in or standard library type behaves otherwise) but strictly speaking that is not a language constraint, so it's not the reference manual's business. Proposed docs patches are always welcome, btw!-) Sep 2, 2010 at 14:29
  • 1
    @Alex: Thank you for the comments. I guess that my confusion comes from the fact that one also says "Unicode string", in Python 2.x: that's why I was wondering whether __repr__() could also return a Unicode string… I have been thinking of submitting doc patches. :) Sep 2, 2010 at 14:50
  • 1
    @EOL, yes, I find string-related terminology ("string", "unicode string", "raw string", ...) unfortunately at risk of ambiguity in common discourse -- I try to always use rigorously non-ambiguous terms such as "str instance", "unicode object", "rawstring literal ", and so forth, but sometimes such rigorous terminology feels stilted in non-formal contexts. In the Language Reference, the only occurrences of the unfortunate "unicode string" are in a single paragraph in 2.4.1 (literals): s/string/object/ there and "string" becomes unambiguous in the Language Reference (where it matters). Sep 2, 2010 at 18:06
  • 3
    It's also possible that the Language Reference is deliberately ambiguous because it's not supposed to be a Reference for CPython only, but for all conforming Python implementations: in Jython and IronPython, which we're very keen to consider fully conforming implementations, all strings are Unicode (and it would be costly and totally against their respective platforms to make things otherwise). Maybe we do need a supplemental CPython implementation-specific reference, as an addition to the implementation-neutral Language one. Sep 2, 2010 at 18:09
  • 1
    Of course, str('<{}>').format(repr(x)) would also work... See stackoverflow.com/questions/809796/…
    – unutbu
    Sep 3, 2010 at 12:04
6

I think a decorator can manage __repr__ incompatibilities in a sane way. Here's what i use:

from __future__ import unicode_literals, print_function
import sys

def force_encoded_string_output(func):

    if sys.version_info.major < 3:

        def _func(*args, **kwargs):
            return func(*args, **kwargs).encode(sys.stdout.encoding or 'utf-8')

        return _func

    else:
        return func


class MyDummyClass(object):

    @force_encoded_string_output
    def __repr__(self):
        return 'My Dummy Class! \N{WHITE SMILING FACE}'
3
  • Nice decorator; I modified it, though, so that _func is not defined when it is not needed. So, __repr__ in Python 2 can apparently return a Unicode string, according to your code (maybe because of unicode_literals?). This clashes with unutbu answer… I find the documentation ambiguous, on this (docs.python.org/2/reference/datamodel.html#object.__repr__, docs.python.org/2/reference/lexical_analysis.html#index-14). I would be interested in any reference information on this, just to be sure that no unforeseen problems can arise from having __repr__ return a Unicode string. Dec 13, 2012 at 3:27
  • @EOL So, __repr__ in Python 2 can apparently return a Unicode string (...) Why do you think so? Jul 10, 2013 at 12:41
  • Good catch, my bad. I will delete my earlier comment, as it is not relevant. Jul 10, 2013 at 15:15
1

I use a function like the following:

def stdout_encode(u, default='UTF8'):
    if sys.stdout.encoding:
        return u.encode(sys.stdout.encoding)
    return u.encode(default)

Then my __repr__ functions look like this:

def __repr__(self):
    return stdout_encode(u'<MyClass {0} {1}>'.format(self.abcd, self.efgh))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.