4

When I try to redirect a UTF-8 stdout in Powershell by running python3 .\test.py > test.txt, error occurs :

UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>

The code in test.py are as following and that is encoded by UTF-8.

print("\u2714")

The code is uploaded here. https://github.com/lingsongfeng/pytest

8
  • Could you provide a reproducible example code?
    – Rivers
    Commented Feb 27, 2021 at 15:49
  • @Rivers OK, I uploaded it on GitHub.
    – flsflsfls
    Commented Feb 27, 2021 at 16:00
  • github.com/lingsongfeng/pytest
    – flsflsfls
    Commented Feb 27, 2021 at 16:02
  • Please post it here (edit your answer). Could you take some time to read this: stackoverflow.com/help/how-to-ask
    – Rivers
    Commented Feb 27, 2021 at 16:05
  • @Rivers Thank you to mention that. I have already edited my question.
    – flsflsfls
    Commented Feb 27, 2021 at 16:16

2 Answers 2

5

When redirecting I/O Python uses a default encoding for Windows (cp1252 for US Windows), but will look to an environment variable if you want to override it:

C:\> set PYTHONIOENCODING=utf8
C:\> test.py > out.txt

Recently, set PYTHONUTF8=1 will also make Python default to UTF-8 for files and I/O redirection.

3
  • 1
    I tried modify the environment variable but it doesn't work. But I found sys.stdout.reconfigure(encoding='utf-8') can help. After adding this line, the redirection can work. However, the output file is UTF-16 LE format with FF FE at the beginning when I hexdumped it.
    – flsflsfls
    Commented Feb 28, 2021 at 3:51
  • It works. How did you test it? I set in cmd window, tested, then started power shell and tested again. Commented Feb 28, 2021 at 11:36
  • Oh that's my fault. I directly copied the CMD set command into Powershell without knowing setting environment variable in pwsh is different... But there is still problem. The output file is formatted by UTF-16 LE, not UTF-8. Neither vscode nor notepad display non ASCII characters correctly.
    – flsflsfls
    Commented Feb 28, 2021 at 13:41
1

As mentioned in python official docs:

On Windows, UTF-8 is used for the console device. Non-character devices such as disk files and pipes use the system locale encoding (i.e. the ANSI codepage). Non-console character devices such as NUL (i.e. where isatty() returns True) use the value of the console input and output codepages at startup, respectively for stdin and stdout/stderr. This defaults to the system locale encoding if the process is not initially attached to a console.

The special behaviour of the console can be overridden by setting the environment variable PYTHONLEGACYWINDOWSSTDIO before starting Python. In that case, the console codepages are used as for any other character device.

Under all platforms, you can override the character encoding by setting the PYTHONIOENCODING environment variable before starting Python or by using the new -X utf8 command line option and PYTHONUTF8 environment variable. However, for the Windows console, this only applies when PYTHONLEGACYWINDOWSSTDIO is also set.

It says that python use UTF-8 for console printing but ANSI for pipes.And IO redirection in powershell is actually a kind of pipes.So some utf-8 characters can't be encoded by ANSI,which causes the error. There are three solutions:

by environment

$env:PYTHONIOENCODING="UTF-8"

by hard code

sys.stdout.reconfigure(encoding='utf-8')

by command

python -X utf8 test.py

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.