103

I've recently changed my program's directory layout: before, I had all my modules inside the "main" folder. Now, I've moved them into a directory named after the program, and placed an __init__.py there to make a package.

Now I have a single .py file in my main directory that is used to launch my program, which is much neater.

Anyway, trying to load in pickled files from previous versions of my program is failing. I'm getting, "ImportError: No module named tools" - which I guess is because my module was previously in the main folder, and now it's in whyteboard.tools, not simply plain tools. However, the code that is importing in the tools module lives in the same directory as it, so I doubt there's a need to specify a package.

So, my program directory looks something like this:

whyteboard-0.39.4

-->whyteboard.py

-->README.txt

-->CHANGELOG.txt

---->whyteboard/

---->whyteboard/__init__.py

---->whyteboard/gui.py

---->whyteboard/tools.py

whyteboard.py launches a block of code from whyteboard/gui.py, that fires up the GUI. This pickling problem definitely wasn't happening before the directory re-organizing.

1
  • perhaps you can add your module to pythonpath (sys.path.append(path_to_your_module)) before pickle load?
    – Yibo Yang
    Commented Aug 14, 2016 at 21:01

8 Answers 8

153

As pickle's docs say, in order to save and restore a class instance (actually a function, too), you must respect certain constraints:

pickle can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored

whyteboard.tools is not the "the same module as" tools (even though it can be imported by import tools by other modules in the same package, it ends up in sys.modules as sys.modules['whyteboard.tools']: this is absolutely crucial, otherwise the same module imported by one in the same package vs one in another package would end up with multiple and possibly conflicting entries!).

If your pickle files are in a good/advanced format (as opposed to the old ascii format that's the default only for compatibility reasons), migrating them once you perform such changes may in fact not be quite as trivial as "editing the file" (which is binary &c...!), despite what another answer suggests. I suggest that, instead, you make a little "pickle-migrating script": let it patch sys.modules like this...:

import sys
from whyteboard import tools

sys.modules['tools'] = tools

and then cPickle.load each file, del sys.modules['tools'], and cPickle.dump each loaded object back to file: that temporary extra entry in sys.modules should let the pickles load successfully, then dumping them again should be using the right module-name for the instances' classes (removing that extra entry should make sure of that).

5
  • 1
    I've tried that but why is it not working for me? :/ Commented Oct 9, 2016 at 14:49
  • Dumping after deleting the the sys.modules entry doesn't work. It gives the error of Can't pickle because the module is missing. Is there something I'm missing? Commented Oct 11, 2018 at 3:53
  • 1
    Thanks a lot for this!
    – Kerem T
    Commented Sep 9, 2019 at 21:44
  • Like a charm! Thanks!
    – ben26941
    Commented May 12, 2020 at 8:40
  • 4
    In case this helps anyone else - In my case it moved to a different package but inside a hierarchy, so it moved from my_old_module.h1.h2 to my_new_module.h1.h2 so I had to override both my_old_module to point to my_new_module but also my_old_module.h1.h2 to point to my_new_module.h1.h2
    – Zionsof
    Commented Dec 13, 2020 at 15:07
51

This can be done with a custom "unpickler" that uses find_class():

import io
import pickle


class RenameUnpickler(pickle.Unpickler):
    def find_class(self, module, name):
        renamed_module = module
        if module == "tools":
            renamed_module = "whyteboard.tools"

        return super(RenameUnpickler, self).find_class(renamed_module, name)


def renamed_load(file_obj):
    return RenameUnpickler(file_obj).load()


def renamed_loads(pickled_bytes):
    file_obj = io.BytesIO(pickled_bytes)
    return renamed_load(file_obj)

Then you'd need to use renamed_load() instead of pickle.load() and renamed_loads() instead of pickle.loads().

3
  • 1
    Maybe old, but thanks for this. I had a real headscratcher when I had a similar issue. Commented Dec 18, 2018 at 11:05
  • Thank you for posting this solution. Works with Python 3.7.x. This saved a lot of work. While the solution posted by Ranch works well enough to read classes of "misplaced" modules, this solution allows for programmatic control AND conversion of such,
    – Frelling
    Commented Jan 9, 2019 at 22:44
  • In case anybody wonders here is my full question concerning the np.load. Any hints would be much appreciated.
    – BadAtLaTeX
    Commented Feb 13, 2020 at 12:40
31

Happened to me, solved it by adding the new location of the module to sys.path before loading pickle:

import sys
sys.path.append('path/to/whiteboard')
f = open("pickled_file", "rb")
pickle.load(f)
3
  • what is the value of f ?
    – vikkyhacks
    Commented Dec 12, 2017 at 18:10
  • f is the opened pickle file Commented Aug 6, 2020 at 13:57
  • To any future visitors, note that this approach is not recommended and can have unintended consequences. This creates multiple copies of the same module at different package locations. That can be very bad, since if that module defines a class Foo, then whyteboard.tools.Foo and tools.Foo will be distinct types, and will fail unexpectedly when used in isinstance checks or expect clauses, among other surprising consequences. You should prefer one of the two methods that actually re-map the old module names to their new locations (i.e. using sys.modules or a custom Unpickler) Commented Oct 22, 2023 at 19:04
14

pickle serializes classes by reference, so if you change were the class lives, it will not unpickle because the class will not be found. If you use dill instead of pickle, then you can serialize classes by reference or directly (by directly serializing the class instead of it's import path). You simulate this pretty easily by just changing the class definition after a dump and before a load.

Python 2.7.8 (default, Jul 13 2014, 02:29:54) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> 
>>> class Foo(object):
...   def bar(self):
...     return 5
... 
>>> f = Foo()
>>> 
>>> _f = dill.dumps(f)
>>> 
>>> class Foo(object):
...   def bar(self, x):
...     return x
... 
>>> g = Foo()
>>> f_ = dill.loads(_f)
>>> f_.bar()
5
>>> g.bar(4)
4
2
  • 1
    Has this changed in the meantime? Can't reproduce your example. (was using py3, though)
    – bariod
    Commented Aug 14, 2019 at 12:21
  • 4
    @bariod: Yes, actually... now you need to give ignore=True in the call to loads to use the stored class. If you don't, it references whatever Foo is defined in main if one exists (and only use the stored Foo if a Foo doesn't already exist in main). Commented Aug 15, 2019 at 2:05
4

This is the normal behavior of pickle, unpickled objects need to have their defining module importable.

You should be able to change the modules path (i.e. from tools to whyteboard.tools) by editing the pickled files, as they are normally simple text files.

2

For people like me needing to update lots of pickle dumps, here's a function implementing @Alex Martelli's excellent advice:

import sys
from types import ModuleType
import pickle

# import torch

def update_module_path_in_pickled_object(
    pickle_path: str, old_module_path: str, new_module: ModuleType
) -> None:
    """Update a python module's dotted path in a pickle dump if the
    corresponding file was renamed.

    Implements the advice in https://stackoverflow.com/a/2121918.

    Args:
        pickle_path (str): Path to the pickled object.
        old_module_path (str): The old.dotted.path.to.renamed.module.
        new_module (ModuleType): from new.location import module.
    """
    sys.modules[old_module_path] = new_module

    dic = pickle.load(open(pickle_path, "rb"))
    # dic = torch.load(pickle_path, map_location="cpu")

    del sys.modules[old_module_path]

    pickle.dump(dic, open(pickle_path, "wb"))
    # torch.save(dic, pickle_path)

In my case, the dumps were PyTorch model checkpoints. Hence the commented-out torch.load/save().

Example

from new.location import new_module

for pickle_path in ('foo.pkl', 'bar.pkl'):
    update_module_path_in_pickled_object(
        pickle_path, "old.module.dotted.path", new_module
    )
1

When you try to load a pickle file that contain a class reference, you must respect the same structure when you saved the pickle. If you want use the pickle somewhere else, you have to tell where this class or other object is; so do this below you can save the day:

import sys
sys.path.append('path/to/folder containing the python module')
1
  • 1
    To any future visitors, note that this approach is not recommended and can have unintended consequences. This creates multiple copies of the same module at different package locations. That can be very bad, since if that module defines a class Foo, then whyteboard.tools.Foo and tools.Foo will be distinct types, and will fail unexpectedly when used in isinstance checks or expect clauses, among other surprising consequences. You should prefer one of the two methods that actually re-map the old module names to their new locations (i.e. using sys.modules or a custom Unpickler) Commented Oct 22, 2023 at 19:04
0

I know this has been a while, but this fixed it for me:

Essentially, use full import path (eg. concurrent.run_concurrent) instead of just the module name (eg. run_concurrent)


Shared Code:

import importlib
module_path="concurrent.run_concurrent"

...

module = importlib.util.module_from_spec(spec)

Original (bad):

module_name = module_path.split(".")[-1]

spec = importlib.util.spec_from_file_location(module_name, filepath)

...

sys.modules[module_name] = module

Replace with the following (remove all references to module_name):

# Remove "module_name"

# Use "module_path" instead of "module_name"
spec = importlib.util.spec_from_file_location(module_path, filepath)

...

# Use "module_path" instead of "module_name"
sys.modules[module_path] = module

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.