4

I have a problem loading objects via numpy.load after renaming a module. Here's a simple example showing the problem.

Imagine having a class defined in mymodule.py:

class MyClass(object):
    a = "ciao"
    b = [1, 2, 3]

    def __init__(self, value=2):
        self.value = value

from a python session I can simply create an instance and save it:

import numpy as np
import mymodule

instance = mymodule.MyClass()
np.save("dump.npy", instance)

Loading the file works nicely (even from a fresh session started in the same folder):

np.load("dump.npy")

If I now rename the module:

mv mymodule.py mymodule2.py

the loading fails. This is expected, but I was hoping that by importing the module before loading:

import mymodule2 as mymodule

the object definition could be found ... but it does not work. This means that: 1. I do not understand how it works 2. I am forced to keep a symbolic link to the renamed file in a project I am partially refactoring.

Is there anything else I can do do avoid the symbolic link solution ? and to avoid having the same problem in the future ?

Thanks a lot, marco [this is my first question here, sorry If I am doing something wrong]

1
  • It was quite interesting to me, after renaming a file, I cannot deserialize/load the original pickled objects! May 12, 2019 at 0:29

1 Answer 1

8

NumPy uses pickle for arrays with objects, but adds a header on top of it. Therefore, you'll need to do a bit more than coding a custom Unpickler:

import pickle

from numpy.lib.format import read_magic, _check_version, _read_array_header


class RenamingUnpickler(pickle.Unpickler):
    def find_class(self, module, name):
        if module == 'mymodule':
            module = 'mymodule2'
        return super().find_class(module, name)


with open('dump.npy', 'rb') as fp:
    version = read_magic(fp)
    _check_version(version)
    dtype = _read_array_header(fp, version)[2]
    assert dtype.hasobject
    print(RenamingUnpickler(fp).load())
3
  • Thanks @wrwrwr, the RenamingUnpickler class works for general purpose, not only NumPy pickle. I used the class to recreate pickle objects that was referring renamed modules and classes. Apr 23, 2018 at 20:23
  • What exactely is necessary ontop of coding a custom unpickler? I have a similar problem with keras/tensorflow.keras and would need a little hint (see here please).
    – gr4nt3d
    Feb 13, 2020 at 12:42
  • I know this is quite an old comment, but I have just used it to fix some errors. Can this be extended to an npz file instead of a npy file?
    – dtward
    Mar 1 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.