37

I'm using type() to dynamically generate classes that will ultimately be pickled. The problem is that the un-pickling process needs the definition of the class in order to re-construct the object that has been pickled.

This is where I'm stuck. I don't know how to somehow provide the unpickler a way to generate an instance from a class that was dynamically generated.

Any hints appreciated.

Thanks!

Here's an example of the problem:

    >>> class Foo(object):
    ...     pass
    >>> g=type('Goo',(Foo,),{'run':lambda self,x: 2*x } )()
    >>> cPickle.dumps(g)

    PicklingError: Can't pickle <class '__main__.Goo'>: attribute lookup __main__.Goo failed

This evidently works, but only from dynamic classes created from a pickle-able base class (with find-able module definition):

import cPickle

class Foo(object): pass

def dynamic(): return type('Goo',(Foo,),{'run':lambda self,x: 2*x } )()

g=type('Goo',(Foo,),{'run':lambda self,x: 2*x , '__reduce__': lambda self: (dynamic,tuple()) } )()

gg=cPickle.loads ( cPickle.dumps(g) )
print gg.run(10)
3
  • 3
    Avoid using pickle... it's slow, brittle, and can open up arbitrary code execution vulnerabilities. Instead, use a serialization format like JSON or YAML. Jul 25, 2012 at 21:04
  • Could you provide an example code of your dynamically created class?
    – pyrospade
    Aug 2, 2012 at 3:23
  • 2
    @ColinDunklau i think if you use python's multiprocessing module and shared memory, then your forced to ensure objects your sharing can be pickled. The mp modules uses pickling to achieve shared memory. Jul 24, 2018 at 3:05

6 Answers 6

23
+50

When the Pickler encounters an object of a type it knows nothing about, it looks for a reduce method. Defining this method when you build your custom class using type should solve the problem of pickling.

If you provide initial args then in addition you might need to define a getnewargs method

2
6

You can assign a global name to your dynamically generated class to make it picklable.

>>> class Foo(object):
...     pass
>>> class_name = 'Goo'
>>> my_class = type(class_name, (Foo, ), {'run': lambda self, x: 2*x })
>>> globals()[class_name] = my_class
>>> g = my_class()
>>> pickle.dumps(g)

Of course, you need to make sure that the names of your classes are unique.

2

One idea would be to pickle a tuple with:

  1. The name of the dynamic class
  2. The subclass tuple (possibly in string form from repr())
  3. The class dictionary
  4. The actual instance

This would allow you to pickle a class and then reconstruct it later using type() and subclassing Unpickler.

1

For non-dynamic classes, python's pickling mechanism records the module and class name as strings. At unpickle time, it automatically loads the class object from that module.

As mentioned in the original post, the problem here is that for dynamic classes, the class definition itself is not pickled by the default pickler. Since dynamic classes don't exist in a module's source file, unpickling a dynamic class generally won't work.

The trickiest part of pickling the class itself is storing the methods' bytecode. Buried in PiCloud, there's an enhanced pickler under the hood that can pickle dynamic functions you could probably use or extend it to handle your objects.

2
  • It's in cloud.serialization.cloudpickle
    – UsAaR33
    Dec 12, 2012 at 22:31
  • 1
    there is also a package called dill which can also pickle dynamically created classes and class instances. It's a very small package that can serialize most python objects. Jul 10, 2014 at 3:34
0

I have never done it myself, but http://docs.python.org/library/pickle.html#subclassing-unpicklers seems to indicate that you can override the behavior by subclassing Unpickler

0

Well, for the posterity; works with cloudpickle

import cloudpickle

class Foo(object):
    pass
g=type('Goo',(Foo,),{'run':lambda self,x: 2*x } )()
cloudpickle.dumps(g)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.