3

The following code writes a compressed text file using gzip, bz2 and lzma, then reads and prints its binary content.

import bz2
import gzip
import lzma
import os


def test(encoding):
    print(encoding)
    for module in [gzip, bz2, lzma]:

        path = '/tmp/test.txt.%s' % module.__name__
        if os.path.exists(path):
            os.remove(path)

        with module.open(path, 'wt', encoding=encoding) as fout:
            fout.write('Ciao')

        with module.open(path, 'rb') as fin:
            print("%8s" % module.__name__, 'bytes:', fin.read())


test('utf-16')
print('')
test('utf-32')

The output is:

utf-16
    gzip bytes: b'\xff\xfeC\x00i\x00a\x00o\x00'
     bz2 bytes: b'C\x00i\x00a\x00o\x00'
    lzma bytes: b'C\x00i\x00a\x00o\x00'

utf-32
    gzip bytes: b'\xff\xfe\x00\x00C\x00\x00\x00i\x00\x00\x00a\x00\x00\x00o\x00\x00\x00'
     bz2 bytes: b'C\x00\x00\x00i\x00\x00\x00a\x00\x00\x00o\x00\x00\x00'
    lzma bytes: b'C\x00\x00\x00i\x00\x00\x00a\x00\x00\x00o\x00\x00\x00'

As you can see, bz2 and lzma don't write the BOM (Byte Order Mark), while gzip does as expected. This means that if I attempt to read the bz2/lzma files in text mode (e.g. bz2.open(path, 'rt', encoding='utf-16')), a UnicodeError is raised complaining for the missing BOM.

Why is that? Is it a bug?

5
  • looks like a bug or a limitation. You should use bnary mode. Mar 14, 2019 at 20:34
  • Sure, I was just curious about it. It could be useful for a future reader to know that it's possible to use an incremental encoder for adding the BOM to an encoded string only the first time it is used (str.encode on the other hand adds the BOM every single time): encoder = codecs.getincrementalencoder(encoding)()
    – janluke
    Mar 15, 2019 at 2:16
  • the documentation for gzip & bz2 modules are copy/paste from each other in that section, but it behaves differently with BOM... weird. Mar 15, 2019 at 6:24
  • Yes, and the code works the same. A binary writer is wrapped by a io.TextIOWrapper. Apparently, it's this wrapper that, for some reason, skips the BOM.
    – janluke
    Mar 15, 2019 at 12:27
  • TextIOWrapper is used for both if you check the documentation. Maybe you could try with objects / binary mode and use TextIOWrapper manually. Mar 15, 2019 at 12:31

1 Answer 1

0

I'm answering my own question. In short: yes, it's definitely a bug of the C implementation of io.TextIOWrapper.

When you open a file in text mode (compressed or not) what is returned is a io.TextIOWrapper that wraps a binary file reader. io.TextIOWrapper is implemented in C in the _io extension module. It turns out that there is also a Python implementation of the io module, namely the _pyio module. _pyio.TextIOWrapper works as expected, so it's definitely a bug of the C implementation.

The following code demonstrates the issue:

import bz2
import io
import _pyio

def test(io_module, encoding='utf-16'):
    path = '/tmp/test.txt.bz2'

    with io_module.TextIOWrapper(bz2.open(path, 'w'), encoding=encoding) as fout:
        fout.write('Ciao')

    with bz2.open(path, 'rb') as fin:
        print("%5s" % io_module.__name__, 'bytes:', fin.read())


test(io)
test(_pyio)

which prints:

   io bytes: b'C\x00i\x00a\x00o\x00'
_pyio bytes: b'\xff\xfeC\x00i\x00a\x00o\x00'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.