27

Imagine a base class that you'd like to inherit from:

class Shape:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y

There seem to be two common patterns of handling a parent's kwargs in a child class's __init__ method.

You can restate the parent's interface completely:

class Circle(Shape):
    def __init__(self, x: float, y: float, radius: float):
        super().__init__(x=x, y=y)
        self.radius = radius

Or you can specify only the part of the interface which is specific to the child, and hand the remaining kwargs to the parent's __init__:

class Circle(Shape):
    def __init__(self, radius: float, **kwargs):
        super().__init__(**kwargs)
        self.radius = radius

Both of these seem to have pretty big drawbacks, so I'd be interested to hear what is considered standard or best practice.

The "restate the interface" method is appealing in toy examples like you commonly find in discussions of Python inheritance, but what if we're subclassing something with a really complicated interface, like pandas.DataFrame or logging.Logger?

Also, if the parent interface changes, I have to remember to change all of my child class's interfaces to match, type hints and all. Not very DRY.

In these cases, you're almost certain to go for the **kwargs option.

But the **kwargs option leaves the user unsure about which arguments are actually required.

In the toy example above, a user might naively write:

circle = Circle()  # Argument missing for parameter "radius"

Their IDE (or mypy or Pyright) is being helpful and saying that the radius parameter is required.

circle = Circle(radius=5)

The IDE (or type checker) is now happy, but the code won't actually run:

Traceback (most recent call last):
  File "foo.py", line 13, in <module>
    circle = Circle(radius=5)
  File "foo.py", line 9, in __init__
    super().__init__(**kwargs)
TypeError: Shape.__init__() missing 2 required positional arguments: 'x' and 'y'

So I'm stuck with a choice between writing out the parent interface multiple times, and not being warned by my IDE when I'm using a child class incorrectly.

What to do?

Research

This mypy issue is loosely related to this.

This reddit thread has a good rehearsal of the relevant arguments for/against each approach I outline.

This SO question is maybe a duplicate of this one. Does the fact I'm talking about __init__ make any difference though?

I've found a real duplicate, although the answer is a bit esoteric and doesn't seem like it would qualify as best, or normal, practice.

15
  • To address the concern of the scenario you brought up with more complex classes like Logger, this is where, to reference the article, you would leverage the adapter pattern. You would be protecting your code by creating that interface between what you don't control and setting something you do control that your own code will then leverage. This way it won't matter what does in fact change with code that isn't yours.
    – idjaw
    Commented Oct 8, 2022 at 14:42
  • What IDE are you using? Because I do get warnings for these. I use PyCharm.
    – idjaw
    Commented Oct 8, 2022 at 15:11
  • We can abstract away from IDEs by talking about Mypy, which is a fairly standard type checker. I'll update the question. @idjaw, can you find out which type checker is showing you a warning in Pycharm?
    – LondonRob
    Commented Oct 8, 2022 at 15:31
  • 1
    What you want is not possible, you'd need the type checker to analyze the body of the constructor in order to check the parameters, that's not something type checkers do. If you want to share the kwargs, there have been proposal, e.g. peps.python.org/pep-0692, I don't know if any has been accepted yet but I'm pretty sure none of the type checkers out there support this.
    – Holt
    Commented Oct 11, 2022 at 11:27
  • 3
    Not a canonical answer, but I would tend to repeat the required arguments but leave the optional ones to **kwargs. It's a half solution not digging to the real question, but a decent compromise in my opinion.
    – joanis
    Commented Oct 11, 2022 at 12:36

10 Answers 10

20
+500

If the parent class has required (positional) arguments (as your Shape class does), then I'd argue that you must include those arguments in the __init__ of the child (Circle) for the sake of being able to pass around "shape-like" instances and be sure that a Circle will behave like any other shape. So this would be your Circle class:

class Shape:
    def __init__(x: float, y: float):
        self.x = x
        self.y = y


class Circle(Shape):
    def __init__(x: float, y: float, radius: float):
        super().__init__(x=x, y=y)
        self.radius = radius


# The expectation is that this should work with all instances of `Shape`
def move_shape(shape: Shape, x: float, y: float):
    shape.x = x
    shape.y = y

However if the parent class is using optional kwargs, that's where stuff gets tricky. You shouldn't have to define colour: str on your Circle class just because colour is an optional argument for Shape. It's up to the developer using your Circle class to know the interface of all shapes and if need be, interrogate the code and note that Circle can accept colour=green as it passes **kwargs to its parent constructor:

class Shape:
    def __init__(x: float, y: float, colour: str = "black"):
        self.x = x
        self.y = y
        self.colour = colour 


class Circle(Shape):
    def __init__(x: float, y: float, radius: float, **kwargs):
        super().__init__(x=x, y=y, **kwargs)
        self.radius = radius


def move_shape(shape: Shape, x: float, y: float):
    shape.x = x
    shape.y = y


def colour_shape(shape: Shape, colour: str):
    shape.colour = colour

Generally my attitude is that a docstring exists to explain why something is written the way it is, not what it's doing. That should be clear from the code. So, if your Circle requires an x and y parameter for use in the parent class, then it should say as much in the signature. If the parent class has optional requirements, then **kwargs is sufficient in the child class and it's incumbent upon the developer to interrogate Circle and Shape to see what the options are.

1
  • This is pretty much the same as joanis's answer but a bit more fleshed out. I'm happy to award the bounty here, but an honourable mention and thanks goes to joanis.
    – LondonRob
    Commented Oct 18, 2022 at 17:12
10

The solution I would consider most reasonable (though I realize what I'm saying might not be canonical) is to repeat the parent-class parameters that are required, but leave the optional ones to **kwargs.

Benefits:

  • clean code that is easy for a human reader to understand,
  • keeps the type checkers happy,
  • repeats only the essential stuff,
  • supports all the optional parameters without repeating those.
2
  • I think this is a very sensible compromise. I'm going to upvote it, but I'll leave the bounty to see whether someone can come up with a more canonical answer, including maybe a link to this kind of discussion in a PEP or on some serious Python message board.
    – LondonRob
    Commented Oct 11, 2022 at 12:45
  • Sounds good. I'll be interested to read any alternative answers or canonical references, too.
    – joanis
    Commented Oct 11, 2022 at 12:46
5

TL;DR

In Python 3.10, dataclasses provide a clean solution

Dataclasses provides inheritance mechanisms with automatically generated cumulative constructors that provide everything needed for mypy or vscode to do the type checking, yet is completely DRY.

Starting with Python 3.10, dataclasses are suitable more often than one might think, so I contend they could be considered a canonical answer to the question here, although probably not for earlier versions of Python.

Details

My thought process

I've been doing some more thinking and research on this topic, partly inspired by other answers and comments here. I have also done some tests, and eventually convinced myself that the canonical answer should be to use @dataclass whenever possible (which is more often than one might think, at least with Python >= 3.10).

There will of course be cases that really cannot be cast to dataclasses, and then I'll say use my first answer for those.

Augmented example without dataclasses

Let me augment the example a bit to illustrate my idea.

class Shape:

    def __init__(self, x: float, y: float, name="default name", verbose=False):
        self.x = x
        self.y = y
        self.name = name
        if verbose:
            print("Initialized:", self)

    def __repr__(self):
        return f"{type(self).__name__}(x={self.x},y={self.y},name={self.name})"

class Circle(Shape):

    def __init__(self, x: float, y: float, r: float, **kwargs):
        self.r = r
        super().__init__(x, y, **kwargs)

    def __repr__(self):
        return f"{type(self).__name__}(r={self.r},x={self.x},y={self.y},name={self.name})"

Here I've added the optional parameter name with a default value that gets stored, and the optional parameter verbose that affects what __init__ does without getting stored. Those add parameters to __init__ beyond just required data fields.

And I've already applied the solution I suggested in my first answer, which was to repeat the required arguments, but only the required arguments.

Solution using dataclasses in Python 3.10

Now, let's rewrite this with dataclasses:

from dataclasses import dataclass, InitVar, field

@dataclass
class Shape:
    x: float
    y: float
    name: str = field(kw_only=True, default="default name")
    verbose: InitVar[bool] = field(kw_only=True, default=False)

    def __post_init__(self, verbose):
        if verbose:
            print("Initialized:", self)

@dataclass
class Circle(Shape):
    r: float

Notice how much shorter the code is. name is still optional with a default value. verbose is still accepted as an initialization parameter that is not stored. I get my __repr__ for free. And it makes the constructor of Circle explicitly require x and y, as well as r, so both mypy and pylint (and presumably vscode too) do complain if any of them is missing. In fact, being automatically generated, the Circle constructor repeats everything in the Shape constructor, but I didn't have to write it, so that's perfectly DRY.

An inherited init-only parameter

Let's add another init-only parameter, scale, which has the effect of scaling everything in both classes, i.e., x, y and r are multiplied by scale. This case is a bit twisted, but it lets me require a __post_init__ in the subclass too, which I would like to illustrate.

from dataclasses import dataclass, InitVar, field

@dataclass
class DCShape:
    x: float
    y: float
    scale: InitVar[float] = field(kw_only=True, default=1)
    name: str = field(kw_only=True, default="default name")
    verbose: InitVar[bool] = field(kw_only=True, default=False)

    def __post_init__(self, scale, verbose):
        self.x = scale * self.x
        self.y = scale * self.y
        if verbose:
            print("Initialized:", self)

@dataclass
class DCCircle(DCShape):
    r: float

    def __post_init__(self, scale, verbose):
        self.r = scale * self.r
        super().__post_init__(scale, verbose)

This is a pretty decent solution too, in my opinion. I did have to repeat scale and verbose in both classes' __post_init__ functions, and the subclass's instance has to call the superclass's instance explicitly, but this is still something I'd be happy to use in real production code.

Why not with Python <= 3.9?

To make this clean, I had to use keyword-only fields, which were only introduced to dataclasses with Python 3.10.

With earlier version, I would have had to also give Circle.r a default value, and then presumably add custom code to make sure that default value wasn't use, which would mean mypy would not notice if r was missing, so I feel that kills that solution. Although for cases where the base class only has required field, dataclasses work well before 3.10 too.

References

2
  • I really like this answer. The more I use data classes the more I wonder why all Python classes don't work like this. Presumably it's because you don't control the init, but so what?!
    – LondonRob
    Commented Oct 18, 2022 at 17:16
  • Well, really, the fact that you don't control the init is a feature, really, and now that I found out about __post_init__, it seems you can still do anything you might have wanted to do in __init__ in the first place.
    – joanis
    Commented Oct 18, 2022 at 23:15
4

I think the best way to do this is to use the **kwargs approach, but to also define a __signature__ attribute on the class. This is a typing.Signature object that describes the arguments that the class expects.

from typing import Signature

class Shape:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y

class Circle(Shape):
    def __init__(self, radius: float, **kwargs):
        super().__init__(**kwargs)
        self.radius = radius

    __signature__ = Signature(
        parameters=[
            Parameter('radius', Parameter.POSITIONAL_OR_KEYWORD, annotation=float),
            Parameter('x', Parameter.POSITIONAL_OR_KEYWORD, annotation=float),
            Parameter('y', Parameter.POSITIONAL_OR_KEYWORD, annotation=float),
        ]
    )

This will allow type checkers to understand that radius is a required argument, and that x and y are optional.

7
  • Nice. It would be even better to put x and y in Shape's signature and copy them into Circle's without repeating them.
    – joanis
    Commented Oct 12, 2022 at 11:40
  • On detail, in Circle's signature, x and y are not positional, they're only keyword, even though they were positional in Shape.
    – joanis
    Commented Oct 12, 2022 at 11:41
  • OK, I take it back: this seems like a great solution, but what version of Python does it work with? I get "ImportError: cannot import name 'Signature' from 'typing'" in both 3.9 and 3.10.
    – joanis
    Commented Oct 12, 2022 at 12:25
  • This is cool and interesting, so it gets an upvote from me, but I don't really see how it gets us past the fundamental problem: that each subclass needs to restate the interface of the super class. Not very DRY.
    – LondonRob
    Commented Oct 12, 2022 at 12:44
  • Is it maybe supposed to be from inspect import Signature? docs
    – LondonRob
    Commented Oct 12, 2022 at 12:47
3

I come from C++. In there this is OOP 101. But since I transitioned to Python, throughout my career I used this no matter how annoying it was to duplicate parent constructor arguments. Also I found the hard way it was very hard to debug with **kwargs with a large enough code base. So, based on my experience I will upvote @joanis answer highlighting the benefits

Benefits:

clean code that is easy for a human reader to understand
keeps the type checkers happy
repeats only the essential stuff
supports all the optional parameters without repeating those.

Although if one needs **kwargs is an option, but as far as I know, type checking will still not work; i.e. a debugging nightmare. Python is after all, an interpreted language.

And if you analyse the problem a bit deeper, it makes sense. Python Object creation only looks at a Single Class name eg. r = Circle(x,y) . There is no way to identify what it is inherited from without looking at the class. Where In C++ we could write something like below, which is compiled, and then resolved at runtime. Python IDE's inform most of the error prior to execution. But I can see why its taken time to provide a solution to this particular problem since **kwargs essentially does not indicate any information that could be validated prior to a run.

#C++ code
#include <iostream>    
using namespace std;    
class Shape {                                           
    public:    
    string x = "";      
};     
class Circle: public Shape                      
{      
 public:    
    string x = "50";      
};  
int main(void) {    
     Shape r= Circle();      
    cout<<r.x;     
}    

Do I look forward to this in Python? Well absolutely, along with some proper Polymorphism like C++. But sadly as advertised these languages have very different mechanisms. To quote from here "Python does not have the concept of virtual function or interface like C++. However, Python provides an infrastructure that allows us to build something resembling interfaces’ functionality" (Not a very well reputed site, but I found this quote to be noteworthy)

on the same line a comment on this line in the problem statement;

Also, if the parent interface changes, I have to remember to change all of my child class's interfaces to match, type hints and all. Not very DRY.

I actually believe this is necessary! We should be careful not to change Parent classes after a bunch of code is written.

However, I recommend reading this "Learning Path Advanced Python Programming" by Dr.Gabriele Lanaro (Or any other book regarding Python Design Patterns) to know a bit about how you can avoid pretty much many pitfalls like this via Design Patterns.

Lastly, while this may probably not the complete and satisfactory answer you are looking for my suggestions are;

  1. If the project is large enough, stick to duplicating class constructors without using **kwargs (The solution with signature was nice! I can't see how it can harm at scale too)
  2. If parent class is library file or third party class consider using an Adapter Pattern.
  3. Have look at Builder Pattern and something called Fluent Builder Pattern

Hope this helps!

2

I searched a little bit and this is what I found:

1.-There are lots of libraries that can do the job, like Makefun, or Merge_args, or Python Forge, they all work the same:

Using inspect and/or functools libraries and trying to merge their signatures (they get all parameters by e.g tuple(signature(your_function).parameters.values()), or getfullargspec(your_function.__init__) (remember to slice it) and then replacing the signature. Since they have put a lot effort in that task, I'd recommend you to use them if you want a solid solution.

2.-I had the same problem long ago, and I ended up restating only the most important parameters, and leaving the rest with **kwargs. But there's a better tournaround if you want something complete without any library (but not so DRY by my part haha): just use print(signature(Shape.__init__)) (Remember to from inspect import signature) and copy what's useful to you :).

3.-I saw @cactusboat's answer and I came up to this too, hope it helps:

from inspect import signature, Signature
from itertools import chain
from collections import OrderedDict


def make_signature(subclass, superclass):
    """Returns a signature object for the subclass constructor's __signature__ attribute."""
    sub_params = signature(superclass.__init__).parameters
    sup_params = signature(subclass.__init__).parameters
    mixed_params = OrderedDict(chain(sub_params.items(), sup_params.items()))
    mixed_signature = Signature(parameters=mixed_params.values())
    return mixed_signature


class Shape:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y


class Circle(Shape):
    def __init__(self, radius: float, **kwargs):
        super().__init__(**kwargs)
        self.radius = radius


Circle.__signature__ = make_signature(Circle, Shape)

if __name__ == "__main__":
    help(Circle)  # Circle(self, x: float, y: float, radius: float, **kwargs) ...

4.-As you can see, there are many ways, but there isn't any canonical one, the closest one could be PEP 362, and it has an example on how to Visualize Callable Objects’ Signature. But it's hard, since you would be falling into the adapter pattern.

Hope it helps! I'm kinda new into programming, so I did my best to find the best that I could. Greetings!

2

I believe you have to answer this question (which calls for opinions) from the point of view that you intend your code to be used.

For you as a developer that know the classes that you are coding, **kwargs can save some lengthy copy/pasting and refactoring if a superclass is modified. Meanwhile, you can use the method that you like most and even mix usage because you have full control over your own code. In the end, this is a matter of convenience.

For dev-users that will use your set of classes as a library, they will expect a fully documented API and your own potential refactoring work do not matter to them. So in that case it will be more intuitive to them to have the full parameter list.

For lambda-users that will just use your code without knowing what is inside it, it does not matter at all, but it should work no matter what. A general rule of thumb is to catch as many errors in the IDE while you can still fix it, not at runtime when it is too late. In that regard, **kwargs is more sensitive and is more prone to lead to bad user experience.

1

I would suggest using Pydantic. This introduces a dependency which might be a deal breaker, but I think it might be what you are looking for.

Example:

from Pydantic import BaseModel
class ShapePydantic(BaseModel):
  x: float
  y: float

class CirclePydantic(ShapePydantic):
  radius:float

Type hint: Type hint from inherited class

Worth noting that Pydantic allows extra inputs extra fields (or arguments) by default, but this can be turned off by using extra=Extra.forbid.

from pydantic import Extra
class CirclePydantic(ShapePydantic, extra=Extra.forbid):
  radius:float
3
  • 6
    I think dataclasses.dataclass offers this too, without an external dependency.
    – LondonRob
    Commented Oct 11, 2022 at 12:28
  • 2
    @Oddaspa this is a nice solution for data classes, but doesn't answer the higher-level question, which remains relevant because not all classes are data classes.
    – joanis
    Commented Oct 11, 2022 at 12:32
  • I'm still thinking about this question... I like how both Pydantic's BaseModel and dataclasses provide all this nice automation with no unnecessary repetition, and I wish the solution could work with more cases. I just tested what happens if you try to add a parameter to __init__ that's not just a member variable, you have to write the whole __init__ yourself and repeat everything. But, the __post_init__ can be used to do additional initialization without losing the generated __init__ method, so that might still be a powerful solution.
    – joanis
    Commented Oct 15, 2022 at 4:00
1

It's a bit tricky and depends on how you intend for your class to be used. For example if you allow usage of positional arguments then you get situation that you describe and basically break the expected order of arguments.

My opinion is that if you use **kwargs then it's better to prohibit use of positional arguments at all (notice asterisk before arguments):

class Shape:
    def __init__(self, *, x: float, y: float):
        self.x = x
        self.y = y


class Circle(Shape):
    def __init__(self, *, radius: float, **kwargs):
        super().__init__(**kwargs)
        self.radius = radius

This solves the issue of unexpected order but does not help the end user.

For that I would suggest to use stubs. It's still can be considered a code duplication, although it can be generated for you (granted you have not too complicated code) and if needed can be tweaked manually. Besides that, it allows developer to provide better type annotations in complicated situations.

That way, you can actually even use any variant you like as a developer, as long as stubs match your implementation, even support overloaded initializers and IDE will show to the user which signature is applicable to their arguments.

Still, I would suggest not to mix named positional arguments and **kwargs unless there is a good clear reason for it (like generic decorators or some kind of proxy). Especially complicated things become when using *args, **kwargs combo, since client now can unknowingly pass in the same argument twice (if there are no type annotations/stubs). This forces you to handle such cases and write complicated "parsing" of arguments. Such an approach can be justified in a large and complicated interface and considered better, since, in a way, provides more flexibility, but for a small interface it would be an overkill and pain.

If using *args, **kwargs then stub file is a must in my opinion.

0

This is probably not be the best practice, but if you want to avoid using the docstrings and want the IDE (type checker) to find these errors, you can use this implementation.

class Shape:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y


class Circle:
    def __init__(self, radius: float, shape: Shape):
        self.shape = shape
        self.radius = radius

enter image description here

2
  • 2
    This breaks the polymorphism, so a Circle is no longer a Shape, it just contains one, and any class that expects a Shape object will refuse a Circle. So this is not a solution.
    – joanis
    Commented Oct 11, 2022 at 13:20
  • @joanis Yes, that is right, although, for an alternative approach, any class that expects a shape can be provided with the shape object of that class. I think it might be a solution for that.
    – Divyessh
    Commented Oct 11, 2022 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.