5

Had a quick question regarding a pandas DataFrame and the pd.read_pickle() function. Basically, I have a large but simple Dataframe (333 mb). When I run pd.read_pickle on the dataframe, I am getting and EOFError.

Is there any way around this issue? What might be causing this?

1
  • 6
    When I encountered this error I worked out that it was due to the initial pickling not having completed correctly. The pickle file was created, but not finished correctly. Seems to me this is the only possible source of the EOFError in pickle, that the pickle is malformed, i.e. not finished.
    – greg_data
    May 10, 2017 at 15:05

2 Answers 2

0

I saw the same EOFError when I created a pickle using:

pandas.DataFrame.to_pickle('path.pkl', compression='bz2')

and then tried to read with:

pandas.read_pickle('path.pkl')

I fixed the issue by supplying the compression on read:

pandas.read_pickle('path.pkl', compression='bz2')

According to the Pandas docs:

compression : {‘infer’, ‘gzip’, ‘bz2’, ‘zip’, ‘xz’, None}, default ‘infer’

    string representing the compression to use in the output file. By default, 
    infers from the file extension in specified path.

Thus, simply changing the path from 'path.pkl' to 'path.bz2' also fixed the problem.

0

I can confirm the valuable comment of greg_data:

When I encountered this error I worked out that it was due to the initial pickling not having completed correctly. The pickle file was created, but not finished correctly. Seems to me this is the only possible source of the EOFError in pickle, that the pickle is malformed, i.e. not finished.

My error during pickling was:

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-40-263240bbee7e> in <module>()
----> 1 main()

<ipython-input-38-9b3c6d782a2a> in main()
     43     with open("/content/drive/MyDrive/{}.file".format(tm.id), "wb") as f:
---> 44         pickle.dump(tm, f, pickle.HIGHEST_PROTOCOL)
     45 
     46     print('Coherence:', get_coherence(tm, token_lists, 'c_v'))

TypeError: can't pickle weakref objects

And when reading that pickle file that was obviously not finished during pickling, the reported error occured:

pd.read_pickle(r'/content/drive/MyDrive/TEST_2021_06_01_10_23_02.file')

Error:

---------------------------------------------------------------------------

EOFError                                  Traceback (most recent call last)

<ipython-input-41-460bdd0a2779> in <module>()
----> 1 object = pd.read_pickle(r'/content/drive/MyDrive/TEST_2021_06_01_10_23_02.file')

/usr/local/lib/python3.7/dist-packages/pandas/io/pickle.py in read_pickle(filepath_or_buffer, compression)
    180                 # We want to silence any warnings about, e.g. moved modules.
    181                 warnings.simplefilter("ignore", Warning)
--> 182                 return pickle.load(f)
    183         except excs_to_catch:
    184             # e.g.

EOFError: Ran out of input

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.