7

I haven't figure out how to do pickle load/save's between python 2 and 3 with pandas DataFrames. There is a 'protocol' option in the pickler that I've played with unsuccessfully but I'm hoping someone has a quick idea for me to try. Here is the code to get the error:

python2.7

>>> import pandas; from pylab import *
>>> a = pandas.DataFrame(randn(10,10))
>>> a.save('a2')
>>> a = pandas.DataFrame.load('a2')
>>> a = pandas.DataFrame.load('a3')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/pandas-0.10.1-py2.7-linux-x86_64.egg/pandas/core/generic.py", line 30, in load
    return com.load(path)
  File "/usr/local/lib/python2.7/site-packages/pandas-0.10.1-py2.7-linux-x86_64.egg/pandas/core/common.py", line 1107, in load
    return pickle.load(f)
ValueError: unsupported pickle protocol: 3

python3

>>> import pandas; from pylab import *
>>> a = pandas.DataFrame(randn(10,10))
>>> a.save('a3')
>>> a = pandas.DataFrame.load('a3')
>>> a = pandas.DataFrame.load('a2')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.3/site-packages/pandas-0.10.1-py3.3-linux-x86_64.egg/pandas/core/generic.py", line 30, in load
    return com.load(path)
  File "/usr/local/lib/python3.3/site-packages/pandas-0.10.1-py3.3-linux-x86_64.egg/pandas/core/common.py", line 1107, in load
    return pickle.load(f)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf4 in position 0: ordinal not in range(128)

Maybe expecting pickle to work between python version is a bit optimistic?

2
  • 2
    I think this is to do with the DEFAULT_PROTOCOL of pickle which is backwards incompatible in python 3. Not sure how to remedy this, either can set this value globally or maybe it should be made possible to pass as an argument in save/load. Jan 29, 2013 at 18:12
  • Hmmm, after playing around it seems that it may by something to do with numpy. I can pickle a list from 2 to 3 using the protocal setting but not a numpy.array.
    – mathtick
    Jan 30, 2013 at 3:45

3 Answers 3

8

I had the same problem. You can change the protocol of the dataframe pickle file with the following function in python3:

import pickle
def change_pickle_protocol(filepath,protocol=2):
    with open(filepath,'rb') as f:
        obj = pickle.load(f)
    with open(filepath,'wb') as f:
        pickle.dump(obj,f,protocol=protocol)

Then you should be able to open it in python2 no problem.

1

If somebody uses pandas.DataFrame.to_pickle() then do the following modification in source code to have the capability of pickle protocol setting:

1) In source file /pandas/io/pickle.py (before modification copy the original file as /pandas/io/pickle.py.ori) search for the following lines:

def to_pickle(obj, path):

pkl.dump(obj, f, protocol=pkl.HIGHEST_PROTOCOL)

Change these lines to:

def to_pickle(obj, path, protocol=pkl.HIGHEST_PROTOCOL):

pkl.dump(obj, f, protocol=protocol)

2) In source file /pandas/core/generic.py (before modification copy the original file as /pandas/core/generic.py.ori) search for the following lines:

def to_pickle(self, path):

return to_pickle(self, path)

Change these lines to:

def to_pickle(self, path, protocol=None):

return to_pickle(self, path, protocol)

3) Restart your python kernel if it runs then save your dataframe using any available pickle protocol (0, 1, 2, 3, 4):

# Python 2.x can read this
df.to_pickle('my_dataframe.pck', protocol=2)

# protocol will be the highest (4), Python 2.x can not read this
df.to_pickle('my_dataframe.pck')

4) After pandas upgrade, repeat step 1 & 2.

5) (optional) Ask the developers to have this capability in official releases (because your code will throw exception on any other Python environments without these changes)

Nice day!

1

You can override the highest protocol available for the pickle package:

import pickle as pkl
import pandas as pd
if __name__ == '__main__':
    # this constant is defined in pickle.py in the pickle package:"
    pkl.HIGHEST_PROTOCOL = 2
    # 'foo.pkl' was saved in pickle protocol 4
    df = pd.read_pickle(r"C:\temp\foo.pkl")

    # 'foo_protocol_2' will be saved in pickle protocol 2 
    # and can be read in pandas with Python 2
    df.to_pickle(r"C:\temp\foo_protocol_2.pkl")

This is definitely not an elegant solution but it does the work without changing pandas code directly.

UPDATE: I found that the newer version of pandas, allow to specify the pickle version in the .to_pickle function: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_pickle.html[1] DataFrame.to_pickle(path, compression='infer', protocol=4)

3
  • Hmm didn't work for me, it didn't affect the protocol when it saved for some reason.
    – Kyle
    Apr 13, 2018 at 1:55
  • 1
    @Kyle, I don't know why id doesn't work, but I found that the newer version of pandas, allow to specify the pickle version int the .to_pickle function: pandas.pydata.org/pandas-docs/stable/generated/… ` DataFrame.to_pickle(path, compression='infer', protocol=4)[source]¶ `
    – ot226
    Apr 15, 2018 at 8:58
  • That's great, you should add it as a new answer, as that's the best solution in this situation I think
    – Kyle
    Apr 16, 2018 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.