76

When discussing metaclasses, the docs state:

You can of course also override other class methods (or add new methods); for example defining a custom __call__() method in the metaclass allows custom behavior when the class is called, e.g. not always creating a new instance.

[Editor's note: This was removed from the docs in 3.3. It's here in 3.2: Customizing class creation]

My questions is: suppose I want to have custom behavior when the class is called, for example caching instead of creating fresh objects. I can do this by overriding the __new__ method of the class. When would I want to define a metaclass with __call__ instead? What does this approach give that isn't achievable with __new__?

2
  • 3
    for anyone going in the docs, the statement is unfortunately no where to be found. Commented Apr 3, 2021 at 22:30
  • 2
    @Marine It was removed in 3.3. Here it is in 3.2: Customizing class creation
    – wjandrea
    Commented Jan 17, 2022 at 21:10

6 Answers 6

48

The subtle differences become a bit more visible when you carefully observe the execution order of these methods.

class Meta_1(type):
    def __call__(cls, *a, **kw):
        print "entering Meta_1.__call__()"
        rv = super(Meta_1, cls).__call__(*a, **kw)
        print "exiting Meta_1.__call__()"
        return rv

class Class_1(object):
    __metaclass__ = Meta_1
    def __new__(cls, *a, **kw):
        print "entering Class_1.__new__()"
        rv = super(Class_1, cls).__new__(cls, *a, **kw)
        print "exiting Class_1.__new__()"
        return rv

    def __init__(self, *a, **kw):
        print "executing Class_1.__init__()"
        super(Class_1,self).__init__(*a, **kw)

Note that the code above doesn't actually do anything other than log what we're doing. Each method defers to its parent implementation i.e. its default. So beside logging it's effectively as if you had simply declared things as follows:

class Meta_1(type): pass
class Class_1(object):
    __metaclass__ = Meta_1

And now let's create an instance of Class_1

c = Class_1()
# entering Meta_1.__call__()
# entering Class_1.__new__()
# exiting Class_1.__new__()
# executing Class_1.__init__()
# exiting Meta_1.__call__()

Therefore if type is the parent of Meta_1 we can imagine a pseudo implementation of type.__call__() as such:

class type:
    def __call__(cls, *args, **kwarg):

        # ... a few things could possibly be done to cls here... maybe... or maybe not...

        # then we call cls.__new__() to get a new object
        obj = cls.__new__(cls, *args, **kwargs)

        # ... a few things done to obj here... maybe... or not...

        # then we call obj.__init__()
        obj.__init__(*args, **kwargs)

        # ... maybe a few more things done to obj here

        # then we return obj
        return obj

Notice from the call order above that Meta_1.__call__() (or in this case type.__call__()) is given the opportunity to influence whether or not calls to Class_1.__new__() and Class_1.__init__() are eventually made. Over the course of its execution Meta_1.__call__() could return an object that hasn't even been touched by either. Take for example this approach to the singleton pattern:

class Meta_2(type):
    __Class_2_singleton__ = None
    def __call__(cls, *a, **kw):
        # if the singleton isn't present, create and register it
        if not Meta_2.__Class_2_singleton__:
            print "entering Meta_2.__call__()"
            Meta_2.__Class_2_singleton__ = super(Meta_2, cls).__call__(*a, **kw)
            print "exiting Meta_2.__call__()"
        else:
            print ("Class_2 singleton returning from Meta_2.__call__(), "
                    "super(Meta_2, cls).__call__() skipped")
        # return singleton instance
        return Meta_2.__Class_2_singleton__

class Class_2(object):
    __metaclass__ = Meta_2
    def __new__(cls, *a, **kw):
        print "entering Class_2.__new__()"
        rv = super(Class_2, cls).__new__(cls, *a, **kw)
        print "exiting Class_2.__new__()"
        return rv

    def __init__(self, *a, **kw):
        print "executing Class_2.__init__()"
        super(Class_2, self).__init__(*a, **kw)

Let's observe what happens when repeatedly trying to create an object of type Class_2

a = Class_2()
# entering Meta_2.__call__()
# entering Class_2.__new__()
# exiting Class_2.__new__()
# executing Class_2.__init__()
# exiting Meta_2.__call__()

b = Class_2()
# Class_2 singleton returning from Meta_2.__call__(), super(Meta_2, cls).__call__() skipped

c = Class_2()
# Class_2 singleton returning from Meta_2.__call__(), super(Meta_2, cls).__call__() skipped

print a is b is c
True

Now observe this implementation using a class' __new__() method to try to accomplish the same thing.

import random
class Class_3(object):

    __Class_3_singleton__ = None

    def __new__(cls, *a, **kw):
        # if singleton not present create and save it
        if not Class_3.__Class_3_singleton__:
            print "entering Class_3.__new__()"
            Class_3.__Class_3_singleton__ = rv = super(Class_3, cls).__new__(cls, *a, **kw)
            rv.random1 = random.random()
            rv.random2 = random.random()
            print "exiting Class_3.__new__()"
        else:
            print ("Class_3 singleton returning from Class_3.__new__(), "
                   "super(Class_3, cls).__new__() skipped")

        return Class_3.__Class_3_singleton__ 

    def __init__(self, *a, **kw):
        print "executing Class_3.__init__()"
        print "random1 is still {random1}".format(random1=self.random1)
        # unfortunately if self.__init__() has some property altering actions
        # they will affect our singleton each time we try to create an instance 
        self.random2 = random.random()
        print "random2 is now {random2}".format(random2=self.random2)
        super(Class_3, self).__init__(*a, **kw)

Notice that the above implementation even though successfully registering a singleton on the class, does not prevent __init__() from being called, this happens implicitly in type.__call__() (type being the default metaclass if none is specified). This could lead to some undesired effects:

a = Class_3()
# entering Class_3.__new__()
# exiting Class_3.__new__()
# executing Class_3.__init__()
# random1 is still 0.282724600824
# random2 is now 0.739298365475

b = Class_3()
# Class_3 singleton returning from Class_3.__new__(), super(Class_3, cls).__new__() skipped
# executing Class_3.__init__()
# random1 is still 0.282724600824
# random2 is now 0.247361634396

c = Class_3()
# Class_3 singleton returning from Class_3.__new__(), super(Class_3, cls).__new__() skipped
# executing Class_3.__init__()
# random1 is still 0.282724600824
# random2 is now 0.436144427555

d = Class_3()
# Class_3 singleton returning from Class_3.__new__(), super(Class_3, cls).__new__() skipped
# executing Class_3.__init__()
# random1 is still 0.282724600824
# random2 is now 0.167298405242

print a is b is c is d
# True
4
  • This is a great answer. In your Meta_1.__call__, you have rv = super(Meta_1, cls).__call__(*a, **kw). Can you explain why Meta_1 is the first argument in super ?? Commented Jun 20, 2019 at 19:40
  • Thank you for the answer. I've used part of the sample code and asked a specific question that I was confused about. I feel much better on this topic now. For your reference the question is here: stackoverflow.com/questions/56691487/… Commented Jun 23, 2019 at 20:20
  • Do you mind if I paraphrase your comments and post as an answer to my question: stackoverflow.com/questions/56691487/… ?? Or even better, do you mind spending a minute to copy your comments here and paste as answer to the linked question? I'll give upvotes for sure. Commented Jun 23, 2019 at 20:47
  • So, I thought that super(arg1, arg2) will look into the MRO of the second input argument to find the first input argument, and return the next class to it. But rv = super(Meta_1, cls).__call__(*a, **kw), the MRO for 2nd argument(cls, or Class_1 ), does not contain the 1st input argument(Meta_1), you cannot find Meta_1 in the MRO for Class_1. so I didn't see why would it take us to invoke type.__call__(Class_1). That's why I asked. Commented Jun 23, 2019 at 21:15
47

The direct answer to your question is: when you want to do more than just customize instance creation, or when you want to separate what the class does from how it's created.

See my answer to Creating a singleton in Python and the associated discussion.

There are several advantages.

  1. It allows you to separate what the class does from the details of how it's created. The metaclass and class are each responsible for one thing.

  2. You can write the code once in a metaclass, and use it for customizing several classes' call behavior without worrying about multiple inheritance.

  3. Subclasses can override behavior in their __new__ method, but __call__ on a metaclass doesn't have to even call __new__ at all.

  4. If there is setup work, you can do it in the __new__ method of the metaclass, and it only happens once, instead of every time the class is called.

There are certainly lots of cases where customizing __new__ works just as well if you're not worried about the single responsibility principle.

But there are other use cases that have to happen earlier, when the class is created, rather than when the instance is created. It's when these come in to play that a metaclass is necessary. See What are your (concrete) use-cases for metaclasses in Python? for lots of great examples.

19

One difference is that by defining a metaclass __call__ method you are demanding that it gets called before any of the class's or subclasses's __new__ methods get an opportunity to be called.

class MetaFoo(type):
    def __call__(cls,*args,**kwargs):
        print('MetaFoo: {c},{a},{k}'.format(c=cls,a=args,k=kwargs))

class Foo(object):
    __metaclass__=MetaFoo

class SubFoo(Foo):
    def __new__(self,*args,**kwargs):
        # This never gets called
        print('Foo.__new__: {a},{k}'.format(a=args,k=kwargs))

 sub=SubFoo()
 foo=Foo()

 # MetaFoo: <class '__main__.SubFoo'>, (),{}
 # MetaFoo: <class '__main__.Foo'>, (),{}

Notice that SubFoo.__new__ never gets called. In contrast, if you define Foo.__new__ without a metaclass, you allow subclasses to override Foo.__new__.

Of course, you could define MetaFoo.__call__ to call cls.__new__, but that's up to you. By refusing to do so, you can prevent subclasses from having their __new__ method called.

I don't see a compelling advantage to using a metaclass here. And since "Simple is better than complex", I'd recommend using __new__.

2
  • 3
    Note also that cls.__new__() will get called indirectly if the MetaFoo.__call__() method invokes super(MetaFoo, cls).__call__(*args, **kwargs).
    – martineau
    Commented Feb 14, 2018 at 19:40
  • btw, the metaclass attribute is gone in python3, use class Simple1(object, metaclass = SimpleMeta1): now... gee thanks python-3-patterns-idioms-test.readthedocs.io/en/latest/… Commented Apr 24, 2019 at 21:04
15

I thought a fleshed out Python 3 version of pyroscope's answer might be handy for someone to copy, paste and hack about with (probably me, when I find myself back at this page looking it up again in 6 months). It is taken from this article:

class Meta(type):

     @classmethod
     def __prepare__(mcs, name, bases, **kwargs):
         print('  Meta.__prepare__(mcs=%s, name=%r, bases=%s, **%s)' % (
             mcs, name, bases, kwargs
         ))
         return {}

     def __new__(mcs, name, bases, attrs, **kwargs):
         print('  Meta.__new__(mcs=%s, name=%r, bases=%s, attrs=[%s], **%s)' % (
             mcs, name, bases, ', '.join(attrs), kwargs
         ))
         return super().__new__(mcs, name, bases, attrs)

     def __init__(cls, name, bases, attrs, **kwargs):
         print('  Meta.__init__(cls=%s, name=%r, bases=%s, attrs=[%s], **%s)' % (
             cls, name, bases, ', '.join(attrs), kwargs
         ))
         super().__init__(name, bases, attrs)

     def __call__(cls, *args, **kwargs):
         print('  Meta.__call__(cls=%s, args=%s, kwargs=%s)' % (
             cls, args, kwargs
         ))
         return super().__call__(*args, **kwargs)

print('** Meta class declared')

class Class(metaclass=Meta, extra=1):

     def __new__(cls, myarg):
         print('  Class.__new__(cls=%s, myarg=%s)' % (
             cls, myarg
         ))
         return super().__new__(cls)

     def __init__(self, myarg):
         print('  Class.__init__(self=%s, myarg=%s)' % (
             self, myarg
         ))
         self.myarg = myarg
         super().__init__()

     def __str__(self):
         return "<instance of Class; myargs=%s>" % (
             getattr(self, 'myarg', 'MISSING'),
         )

print('** Class declared')

Class(1)
print('** Class instantiated')

Outputs:

** Meta class declared
  Meta.__prepare__(mcs=<class '__main__.Meta'>, name='Class', bases=(), **{'extra': 1})
  Meta.__new__(mcs=<class '__main__.Meta'>, name='Class', bases=(), attrs=[__module__, __qualname__, __new__, __init__, __str__, __classcell__], **{'extra': 1})
  Meta.__init__(cls=<class '__main__.Class'>, name='Class', bases=(), attrs=[__module__, __qualname__, __new__, __init__, __str__, __classcell__], **{'extra': 1})
** Class declared
  Meta.__call__(cls=<class '__main__.Class'>, args=(1,), kwargs={})
  Class.__new__(cls=<class '__main__.Class'>, myarg=1)
  Class.__init__(self=<instance of Class; myargs=MISSING>, myarg=1)
** Class instantiated

Another great resource highlighted by the same article is David Beazley's PyCon 2013 Python 3 Metaprogramming tutorial.

1
  • If you get it chance it would be good to update this with __init_subclass__ I think. Commented Sep 1 at 23:46
1

It's a matter of lifecycle phases and what you have access to. __call__ gets called after __new__ and is passed the initialization parameters before they get passed on to __init__, so you can manipulate them. Try this code and study its output:

class Meta(type):
    def __new__(cls, name, bases, newattrs):
        print "new: %r %r %r %r" % (cls, name, bases, newattrs,)
        return super(Meta, cls).__new__(cls, name, bases, newattrs)

    def __call__(self, *args, **kw):
        print "call: %r %r %r" % (self, args, kw)
        return super(Meta, self).__call__(*args, **kw)

class Foo:
    __metaclass__ = Meta

    def __init__(self, *args, **kw):
        print "init: %r %r %r" % (self, args, kw)

f = Foo('bar')
print "main: %r" % f
6
  • 2
    No! __new__ on the metaclass happens when the class is created, not an instance. __call__ happens when __new__ would happen without the metaclass.
    – agf
    Commented Aug 6, 2011 at 12:56
  • 1
    Where do I say that __new__ is related to instance creation?
    – pyroscope
    Commented Aug 6, 2011 at 12:59
  • 4
    I was actually asking about the class's __new__, not the metaclass's __new__. Commented Aug 6, 2011 at 13:01
  • It certainly sounds like you're talking about the class' __new__ there rather than the metaclass __new__.
    – agf
    Commented Aug 6, 2011 at 13:03
  • __new__ of a class (not metaclass) is called when the object is created at instantiation of the class. It is useful if you want to return an object that has been created before (e.g. a singleton).instead of re-creating a new object.
    – Nickpick
    Commented Nov 13, 2017 at 16:52
0

In the particular example given in the question, overriding __call__ in the metaclass is just superior to overriding __new__ in the class.

suppose I want to have custom behavior when the class is called, for example caching instead of creating fresh objects

  1. If the purpose of caching is efficiency, then caching __new__'s result is not optimal because __init__ is executed anyhow (Data Model: Basic Customization). For example:

    from functools import lru_cache
    
    class MyClass:
    
        @lru_cache(maxsize=None)
        def __new__(cls, *args, **kwargs):
            return super().__new__(cls)
    
        def __init__(self, ...)
            "Always executed. Even on cache hit."
    
  2. Caching __new__'s result is sound only if __init__ has no noticeable effect on an already initialized instance. Otherwise the execution of __init__ on a cached object could lead to annoying side-effects on its other references.

  3. Caching at the metaclass level avoids both the performance issue of 1. and the correctness issue of 2. For example:

    from functools import lru_cache
    
    class CachedInstances(type):
    
        @lru_cache(maxsize=None)
        def __call__(cls, *args, **kwargs):
            return super().__call__(*args, **kwargs)
    
    class MyClass(metaclass=CachedInstances):
    
        def __init__(self, ...)
            "Only executed on cache miss."
    

Note that Michael Ekoka's answer already mention undesired effects that may arise due to repeated __init__ execution within the override __new__ approach (as in my item 2).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.