6

I have a plain ASCII file. When I try to open it with codecs.open(..., "utf-8"), I am unable to read single characters. ASCII is a subset of UTF-8, so why can't codecs open such a file in UTF-8 mode?

# test.py

import codecs

f = codecs.open("test.py", "r", "utf-8")

# ASCII is supposed to be a subset of UTF-8:
# http://www.fileformat.info/info/unicode/utf8.htm

assert len(f.read(1)) == 1 # OK
f.readline()
c = f.read(1)
print len(c)
print "'%s'" % c
assert len(c) == 1 # fails

# max% p test.py
# 63
# '
# import codecs
#
# f = codecs.open("test.py", "r", "utf-8")
#
# # ASC'
# Traceback (most recent call last):
#   File "test.py", line 15, in <module>
#     assert len(c) == 1 # fails
# AssertionError
# max%

system:

Linux max 4.4.0-89-generic #112~14.04.1-Ubuntu SMP Tue Aug 1 22:08:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Of course it works with regular open. It also works if I remove the "utf-8" option. Also what does 63 mean? That's like the middle of the 3rd line. I don't get it.

16
  • 1
    Also print out the character itself, not just the length, and the byte code for the character.
    – user707650
    Sep 27, 2017 at 0:47
  • 1
    The length suggests it includes the previous readline result as well; interesting.
    – user707650
    Sep 27, 2017 at 0:48
  • 1
    Well, at least I can reproduce this on my Mac, with Python versions 2.7.10, 2.7.13 and 3.6.2. len(c) for me is 59 in all cases.
    – user707650
    Sep 27, 2017 at 0:55
  • 1
    Side-note: Never use codecs.open. It's buggy in weird ways (as @Evert notes, it has to open files in binary mode, which has all sorts of side-effects). Try using io.open instead (it's the same as plain open on Py3, and provides the same interface on Py2), and is both faster and more correct than codecs.open (which is essentially deprecated). I suspect your problem will disappear. Sep 27, 2017 at 1:11
  • 1
    @personal_clown: codecs.open is in a weird state. It's strictly inferior to io.open except in some very unusual cases (bytes<->bytes codecs like ROT13 and hex, as opposed to the standard bytes<->Unicode codecs). Because of the weirdo use cases, they've been reluctant to officially deprecate it, but if you check the Python bug tracker, it's often referred to as pseudo-deprecated. PEP 400 includes a bunch of reasons why StreamReader/StreamWriter/StreamReaderWriter (what codecs.open creates) are broken. Sep 27, 2017 at 1:30

1 Answer 1

5

Found your problem:

When passed an encoding, codecs.open returns a StreamReaderWriter, which is really just a wrapper around (not a subclass of; it's a "composed of" relationship, not inheritance) StreamReader and StreamWriter. Problem is:

  1. StreamReaderWriter provides a "normal" read method (that is, it takes a size parameter and that's it)
  2. It delegates to the internal StreamReader.read method, where the size argument is only a hint as to the number of bytes to read, but not a limit; the second argument, chars, is a strict limiter, but StreamReaderWriter never passes that argument along (it doesn't accept it)
  3. When size hinted, but not capped using chars, if StreamReader has buffered data, and it's large enough to match the size hint StreamReader.read blindly returns the contents of the buffer, rather than limiting it in any way based on the size hint (after all, only chars imposes a maximum return size)

The API of StreamReader.read and the meaning of size/chars for the API is the only documented thing here; the fact that codecs.open returns StreamReaderWriter is not contractual, nor is the fact that StreamReaderWriter wraps StreamReader, I just used ipython's ?? magic to read the source code of the codecs module to verify this behavior. But documented or not, that's what it's doing (feel free to read the source code for StreamReaderWriter, it's all Python level, so it's easy).

The best solution is to switch to io.open, which is faster and more correct in every standard case (codecs.open supports the weirdo codecs that don't convert between bytes [Py2 str] and str [Py2 unicode], but rather, handle str to str or bytes to bytes encodings, but that's an incredibly limited use case; most of the time, you're converting between bytes and str). All you need to do is import io instead of codecs, and change the codecs.open line to:

f = io.open("test.py", encoding="utf-8")

The rest of your code can remain unchanged (and will likely run faster to boot).

As an alternative, you could explicitly bypass StreamReaderWriter to get the StreamReader's read method and pass the limiting argument directly, e.g. change:

c = f.read(1)

to:

# Pass second, character limiting argument after size hint
c = f.reader.read(6, 1)  # 6 is sort of arbitrary; should ensure a full char read in one go

I suspect Python Bug #8260, which covers intermingling readline and read on codecs.open created file objects, applies here, officially, it's "fixed", but if you read the comments, the fix wasn't complete (and may not be possible to complete given the documented API); arbitrarily weird combinations of read and readline will be able to break it.

Again, just use io.open; as long as you're on Python 2.6 or higher, it's available, and it's just plain better.

2
  • Yeah, I really only started seeing my problems right after I started mixing readline and read. Must be the 8260 as you say. Thanks for all the helpful research. I'll take a closer look at io and other options. Sep 27, 2017 at 2:30
  • 1
    @personal_clown: Yeah, readline causes the problem because, to avoid excessive system call overhead, it buffers (72 bytes IIRC). If you just performed read(1) calls, it would read one byte at a time without buffering (because you "hinted" you only needed a byte and it trusts the hint); you'd never get in a position where size vs. chars mattered. But readline prepopulates the buffer, so now size without chars means roughly "return greater of current buffer or size bytes worth of characters". Sep 27, 2017 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.