32

I'm writing a library to access REST API. It returns json with user object. I convert it to dict, and then convert it to dataclass object. The problem is that not all fields are fixed. I want to add additional fields (which are not specified in my dataclass) dynamically. I can simply assign values to my object, but they don't appear in the object representation and dataclasses.asdict function doesn't add them into resulting dict:

from dataclasses import asdict, dataclass

@dataclass
class X:
    i: int

x = X(i=42)
x.s = 'text'

x
# X(i=42)

x.s
# 'text'

asdict(x)
# {'i': 42}

2 Answers 2

43

You could use make_dataclass to create X on the fly:

X = make_dataclass('X', [('i', int), ('s', str)])
x = X(i=42, s='text')

asdict(x)
# {'i': 42, 's': 'text'}

Or as a derived class:

@dataclass
class X:
    i: int

x = X(i=42)
x.__class__ = make_dataclass('Y', fields=[('s', str)], bases=(X,))
x.s = 'text'

asdict(x)
# {'i': 42, 's': 'text'}
3
  • In the case of TypeError: non-default argument 's' follows default argument you should add: x.__class__ = make_dataclass('Y', fields=[('s', str, field(init=False))], bases=(X,))
    – djstrong
    Aug 5, 2020 at 20:11
  • Also, just a note that in the original question, it isn't too clear why a solution with dataclasses is desired. I've actually opted with a slightly different approach, that should be overall much faster than make_dataclass. I've updated my answer below to reflect this as well.
    – rv.kvetch
    Jun 23, 2022 at 15:56
  • going to this effort contradicts the convenience and benefit of dataclasses. they are there to simplify classes and make data structures clear and easy. . . youtube.com/live/0lCIkY32AdI?feature=share
    – D.L
    Oct 19 at 21:41
11

Update (6/22): As it's now mid-2022, I thought I'd refresh my answer with a brand new approach I've been toying around with. I am pleased to announce a fast, modern library I have quite recently released, called dotwiz.

The dotwiz library can be installed with pip:

pip install dotwiz

This is a tiny helper library that I've created, which makes dict objects safe to access by dot notation - such as a.b.c instead of a['b']['c']. From personal tests and benchmarks, it's actually a lot faster than something like make_dataclass - more info on this below.

Additionally, one can also subclass from DotWiz or DotWizPlus, and this enables type hinting and auto-completion hints from an IDE such as PyCharm. Here is a simple example of that below:

from dataclasses import asdict, make_dataclass

from dotwiz import DotWiz


class MyTypedWiz(DotWiz):
    # add attribute names and annotations for better type hinting!
    i: int
    s: str


dw = MyTypedWiz(i=42, s='text')
print(dw)
# ✫(i=42, s='text')

print(dw.to_dict())
# {'i': 42, 's': 'text'}

If you still prefer to use dataclasses to model your data, I've included my original answer below that is mostly unchanged from years past.

Benchmarks

The follow results were timed on a Mac Pro with the M1 chip, Python 3.10.4, and with n=5000 iterations.

Creating or instantiating the object:

$ python -m timeit -n 5000 -s "from dotwiz import DotWiz" -c "DotWiz(i=42, s='text')"
5000 loops, best of 5: 425 nsec per loop

$ python -m timeit -n 5000 -s "from dataclasses import make_dataclass" -c "X = make_dataclass('X', [('i', int), ('s', str)]); X(i=42, s='text')"
5000 loops, best of 5: 97.8 usec per loop

These times are probably inflated, but in this particular case it looks like DotWiz is about 250x faster than make_dataclass. In practice, I would say it's about 100 times faster on average.

Accessing a key by dot notation:

$ python -m timeit -n 5000 -s "from dotwiz import DotWiz" -s "dw = DotWiz(i=42, s='text')" -c "dw.s.lower()"         
5000 loops, best of 5: 39.7 nsec per loop

$ python -m timeit -n 5000 -s "from dataclasses import make_dataclass" -s "X = make_dataclass('X', [('i', int), ('s', str)])" -s "x = X(i=42, s='text')" -c "x.s.lower()"
5000 loops, best of 5: 39.9 nsec per loop

The times to access an attribute or a key look to be mostly the same.

Serializing the object to JSON:

$ python -m timeit -n 5000 -s "import json" -s "from dotwiz import DotWiz" -s "dw = DotWiz(i=42, s='text')" -c "json.dumps(dw)"
5000 loops, best of 5: 1.1 usec per loop

$ python -m timeit -n 5000 -s "import json" -s "from dotwiz import DotWiz" -s "dw = DotWiz(i=42, s='text')" -c "json.dumps(dw.to_dict())"
5000 loops, best of 5: 1.46 usec per loop

$ python -m timeit -n 5000 -s "import json" -s "from dataclasses import asdict, make_dataclass" -s "X = make_dataclass('X', [('i', int), ('s', str)])" -s "x = X(i=42, s='text')" -c "json.dumps(asdict(x))"
5000 loops, best of 5: 2.87 usec per loop

So, it actually looks like it's about 2.5x faster to serialize a DotWiz object, as compared to a dataclass instance.

Original Answer

As mentioned, fields marked as optional should resolve the issue. If not, consider using properties in dataclasses. Yep, regular properties should work well enough - though you'll have to declare field in __post_init__, and that's slightly inconvenient.

If you want to set a default value for the property so accessing getter immediately after creating the object works fine, and if you also want to be able to set a default value via constructor, you can make use of a concept called field properties; a couple libraries like dataclass-wizard provide full support for that.

example usage:

from dataclasses import asdict, dataclass
from typing import Optional

from dataclass_wizard import property_wizard


@dataclass
class X(metaclass=property_wizard):
    i: int
    s: Optional[str] = None

    @property
    def _s(self):
        """Returns a title-cased value, i.e. `stRiNg` -> `String`"""
        return self._s.title() if self._s else None

    @_s.setter
    def _s(self, s: str):
        """Reverses a string, i.e. `olleH` -> `Hello` """
        self._s = s[::-1] if s else None


x = X(i=42)

x
# X(i=42, s=None)

assert x.s is None  # True

x.s = '!emordnilap'

x
# X(i=42, s='Palindrome!')

x.s
# 'Palindrome!'

asdict(x)
# {'i': 42, 's': 'Palindrome!'}

Disclaimer: I am the creator (and maintener) of this library.

2
  • 2
    Adding reference to (claimed) impressive benchmarks (comparing performance to make_dataclass, Box, DotMap, and maybe others): github.com/rnag/dotwiz#benchmarks. (Partial motivation for my writing this comment: I want a web-search hit on dotwiz & dotmap.) Jul 18, 2022 at 19:09
  • dotwiz sounds nice, but really needs to be part of a standard library or method of dataclass as opposed to a satellite import (which gives up convenience). In fact, dataclass should now be a python keyword . . . youtube.com/live/0lCIkY32AdI?feature=share
    – D.L
    Oct 19 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.