33

I am trying to read a gunzipped file (.gz) in python and am having some trouble.

I used the gzip module to read it but the file is encoded as a utf-8 text file so eventually it reads an invalid character and crashes.

Does anyone know how to read gzip files encoded as utf-8 files? I know that there's a codecs module that can help but I can't understand how to use it.

Thanks!

import string
import gzip
import codecs

f = gzip.open('file.gz','r')

engines = {}
line = f.readline()
while line:
    parsed = string.split(line, u'\u0001')

    #do some things...

    line = f.readline()
for en in engines:
  print(en)
3
  • Can you post the code you have so far?
    – Mark Byers
    Dec 10, 2009 at 20:03
  • Can you convert the utf-8 file to ascii then attempt to decompress that? hmm.... Dec 10, 2009 at 20:06
  • If you are getting a UnicodeDecodeError, see this related post, which shows the use of the open('errors') parameter and mentions a caveat when using the ISO-8859-1 (latin-1) encoding: stackoverflow.com/questions/35028683/…
    – Trutane
    Feb 27, 2021 at 10:48

5 Answers 5

55

This is possible since Python 3.3:

import gzip
gzip.open('file.gz', 'rt', encoding='utf-8')

Notice that gzip.open() requires you to explicitly specify text mode ('t').

24
+50

I don't see why this should be so hard.

What are you doing exactly? Please explain "eventually it reads an invalid character".

It should be as simple as:

import gzip
fp = gzip.open('foo.gz')
contents = fp.read() # contents now has the uncompressed bytes of foo.gz
fp.close()
u_str = contents.decode('utf-8') # u_str is now a unicode string

EDITED

This answer works for Python2 in Python3, please see @SeppoEnarvi 's answer at https://stackoverflow.com/a/19794943/610569 (it uses the rt mode for gzip.open.

4
  • +1 ... That is the most lucid and least complicated of the 3 answers so far. Dec 10, 2009 at 22:49
  • 1
    Not necessarily the least complicated, in that you have to decode each line you read. In the getreader implementation, this happens automatically so each line is unicode Jan 5, 2012 at 20:37
  • While it is a nice solution, I have a feeling that this solution won't scale well with large files. Nov 9, 2016 at 15:59
  • Exactly. We want this to be interpreted correctly by the library, not by us after being required to read the whole thing into a string. Jan 17, 2020 at 21:01
21

Maybe

import codecs
zf = gzip.open(fname, 'rb')
reader = codecs.getreader("utf-8")
contents = reader( zf )
for line in contents:
    pass
1
  • 5
    As a one-liner: for line in codecs.getreader('utf-8')(gzip.open(fname), errors='replace') which also adds control over the error handling Jan 5, 2012 at 20:38
7

The above produced tons of decoding errors. I used this:

for line in io.TextIOWrapper(io.BufferedReader(gzip.open(filePath)), encoding='utf8', errors='ignore'):
    ...
0

In pythonic form (2.5 or greater)

from __future__ import with_statement # for 2.5, does nothing in 2.6
from gzip import open as gzopen

with gzopen('foo.gz') as gzfile:
    for line in gzfile:
      print line.decode('utf-8')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.