4

Environment: Windows, Python 3.4.1, 64-bit version.

I tried to save data with pickle and gzip, simply like this:

with gzip.open(filename, 'rb') as f:
    pickle.dump(data,f)

The data can be successfully dumped without gzip, but with gzip, exception raised as:

File "C:\Python34\lib\gzip.py", line 344, in write
  self.fileobj.write( self.compress.compress(data) )
OverflowError: Size does not fit in an unsigned int

I traced back the code, and found that gzip is actually built upon zlib. And after googling this problem, I came across this page http://bugs.python.org/file32715/zlib_64bit-4.patch. There seems a length limit of unsigned int type is imposed.

So, my question is, is there any way to make up for this bug or pass around it?

5
  • Other than performing streaming compression? Nov 6, 2015 at 8:38
  • How big is data and what is the size of an unsigned int on your architecture?
    – Mark Adler
    Nov 6, 2015 at 15:31
  • Looking the C code in Python, that is a rather poor, somewhat lazy response. The code should instead feed UINT_MAX portions of the input until it is all consumed.
    – Mark Adler
    Nov 7, 2015 at 3:24
  • @MarkAdler: The data size is approximately 5.1GB
    – Jedi
    Nov 9, 2015 at 1:22
  • And the size of an int on your machine is, I'm guessing, 32 bits. Then indeed your problem is being caused by a bug in the Python library code. You will have to work around it as augurar has described. However you can use much bigger chunk sizes. In fact, in this specific case, you would only need two chunks.
    – Mark Adler
    Nov 9, 2015 at 1:38

1 Answer 1

1

You could try wrapping the gzip file in a writer that splits the data into chunks of a given maximum size. Here is a sketch:

class ChunkedWriter(object):
    def __init__(self, file, chunksize=65536):
        self.file = file
        self.chunksize = chunksize

    def write(self, data):
        mdata = memoryview(data)
        for i in range(0, len(mdata), self.chunksize):
            self.file.write(bytes(mdata[i:i+self.chunksize]))

I'm not sure whether this will actually solve your problem since I was unable to reproduce it on my own computer.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.