51

I tried using some code like this to read a JSON file (encoded using UTF-8):

input = open("json/world_bank.json")
i=0
for l in input:
    i+=1
print(i)

But I got a UnicodeDecodeError. However, it started working once I tried explicitly specifying an encoding:

input = open("json/world_bank.json",encoding="utf8")

I thought the open function would use "utf8" as the default encoding? Why does it need to be specified?

3
  • 1
    What does sys.getfilesystemencoding() return on your system?
    – marcelm
    Mar 30, 2016 at 9:47
  • here it is 'mbcs' @marcelm Mar 30, 2016 at 9:48
  • Ah hmm, that doesn't tell me too much; could you check open("json/world_bank.json").encoding as well?
    – marcelm
    Mar 30, 2016 at 11:29

2 Answers 2

70

The default UTF-8 encoding of Python 3 only extends to conversions between bytes and str types. open() instead chooses an appropriate default encoding based on the environment:

encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever locale.getpreferredencoding() returns), but any text encoding supported by Python can be used. See the codecs module for the list of supported encodings.

For example, a Windows machine with a Western Europe/North America locale will normally use the 8-bit Windows-1252 character set (Python calls this encoding 'cp1252').

5
  • 16
    Fortunately there are recent attempts to end this madness... someday.
    – Jeyekomon
    Apr 28, 2020 at 14:05
  • 3
    3.9 is installed on my machine and it's still using Windows 1252 encoding. PEP 597 linked by @Jeyekomon now says Python 3.10. Feb 12, 2021 at 8:09
  • 1
    @StefanBerger Guys, read the PEP. It is just about emitting a warning when the encoding argument is not set. They mention they are aware of the encoding problem but the change is far too big. This PEP should ease the process when someday the actual default utf-8 open encoding PEP will be drafted.
    – Jeyekomon
    Feb 14, 2021 at 8:18
  • 3
    This is a good example of a very bad decision that was made a long time ago. Why give the pretense of being cross-platform when things that do not have to be cross-platform are not cross-platform by default. Aug 13, 2021 at 21:57
  • 2
    The madness will finally be ended in Python 3.15. PEP 686: Make UTF-8 mode default has been accepted.
    – Jan Derk
    Aug 6, 2022 at 15:25
1

Following the advice here, the problem can also be solved by setting the environment variable PYTHONUTF8=1. This causes open to use UTF-8 encoding by default, rather than the platform's default encoding.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.