27

I'm building some Python code to read and manipulate deeply nested dicts (ultimately for interacting with JSON services, however it would be great to have for other purposes) I'm looking for a way to easily read/set/update values deep within the dict, without needing a lot of code.

@see also Python: Recursively access dict via attributes as well as index access? -- Curt Hagenlocher's "DotDictify" solution is pretty eloquent. I also like what Ben Alman presents for JavaScript in http://benalman.com/projects/jquery-getobject-plugin/ It would be great to somehow combine the two.

Building off of Curt Hagenlocher and Ben Alman's examples, it would be great in Python to have a capability like:

>>> my_obj = DotDictify()
>>> my_obj.a.b.c = {'d':1, 'e':2}
>>> print my_obj
{'a': {'b': {'c': {'d': 1, 'e': 2}}}}
>>> print my_obj.a.b.c.d
1
>>> print my_obj.a.b.c.x
None
>>> print my_obj.a.b.c.d.x
None
>>> print my_obj.a.b.c.d.x.y.z
None

Any idea if this is possible, and if so, how to go about modifying the DotDictify solution?

Alternatively, the get method could be made to accept a dot notation (and a complementary set method added) however the object notation sure is cleaner.

>>> my_obj = DotDictify()
>>> my_obj.set('a.b.c', {'d':1, 'e':2})
>>> print my_obj
{'a': {'b': {'c': {'d': 1, 'e': 2}}}}
>>> print my_obj.get('a.b.c.d')
1
>>> print my_obj.get('a.b.c.x')
None
>>> print my_obj.get('a.b.c.d.x')
None
>>> print my_obj.get('a.b.c.d.x.y.z')
None

This type of interaction would be great to have for dealing with deeply nested dicts. Does anybody know another strategy (or sample code snippet/library) to try?

3
  • See a much simpler answer in stackoverflow.com/a/53354398/869951
    – Oliver
    Nov 17, 2018 at 18:48
  • You can use the very simple package dict-deep which has the functions deep_get and deep_set. Key can be a string in dotted notation "a.b.c" or anything the list() constructor accepts. [disclaimer: I am the author of dict-deep]
    – mbello
    Feb 27, 2019 at 18:01
  • python-benedict is such a nice library supporting nested dicts and lists.
    – Viljami
    Feb 19, 2021 at 11:59

6 Answers 6

35

Attribute Tree

The problem with your first specification is that Python can't tell in __getitem__ if, at my_obj.a.b.c.d, you will next proceed farther down a nonexistent tree, in which case it needs to return an object with a __getitem__ method so you won't get an AttributeError thrown at you, or if you want a value, in which case it needs to return None.

I would argue that in every case you have above, you should expect it to throw a KeyError instead of returning None. The reason being that you can't tell if None means "no key" or "someone actually stored None at that location". For this behavior, all you have to do is take dotdictify, remove marker, and replace __getitem__ with:

def __getitem__(self, key):
    return self[key]

Because what you really want is a dict with __getattr__ and __setattr__.

There may be a way to remove __getitem__ entirely and say something like __getattr__ = dict.__getitem__, but I think this may be over-optimization, and will be a problem if you later decide you want __getitem__ to create the tree as it goes like dotdictify originally does, in which case you would change it to:

def __getitem__(self, key):
    if key not in self:
        dict.__setitem__(self, key, dotdictify())
    return dict.__getitem__(self, key)

I don't like the marker business in the original dotdictify.

Path Support

The second specification (override get() and set()) is that a normal dict has a get() that operates differently from what you describe and doesn't even have a set (though it has a setdefault() which is an inverse operation to get()). People expect get to take two parameters, the second being a default if the key isn't found.

If you want to extend __getitem__ and __setitem__ to handle dotted-key notation, you'll need to modify doctictify to:

class dotdictify(dict):
    def __init__(self, value=None):
        if value is None:
            pass
        elif isinstance(value, dict):
            for key in value:
                self.__setitem__(key, value[key])
        else:
            raise TypeError, 'expected dict'

    def __setitem__(self, key, value):
        if '.' in key:
            myKey, restOfKey = key.split('.', 1)
            target = self.setdefault(myKey, dotdictify())
            if not isinstance(target, dotdictify):
                raise KeyError, 'cannot set "%s" in "%s" (%s)' % (restOfKey, myKey, repr(target))
            target[restOfKey] = value
        else:
            if isinstance(value, dict) and not isinstance(value, dotdictify):
                value = dotdictify(value)
            dict.__setitem__(self, key, value)

    def __getitem__(self, key):
        if '.' not in key:
            return dict.__getitem__(self, key)
        myKey, restOfKey = key.split('.', 1)
        target = dict.__getitem__(self, myKey)
        if not isinstance(target, dotdictify):
            raise KeyError, 'cannot get "%s" in "%s" (%s)' % (restOfKey, myKey, repr(target))
        return target[restOfKey]

    def __contains__(self, key):
        if '.' not in key:
            return dict.__contains__(self, key)
        myKey, restOfKey = key.split('.', 1)
        target = dict.__getitem__(self, myKey)
        if not isinstance(target, dotdictify):
            return False
        return restOfKey in target

    def setdefault(self, key, default):
        if key not in self:
            self[key] = default
        return self[key]

    __setattr__ = __setitem__
    __getattr__ = __getitem__

Test code:

>>> life = dotdictify({'bigBang': {'stars': {'planets': {}}}})
>>> life.bigBang.stars.planets
{}
>>> life.bigBang.stars.planets.earth = { 'singleCellLife' : {} }
>>> life.bigBang.stars.planets
{'earth': {'singleCellLife': {}}}
>>> life['bigBang.stars.planets.mars.landers.vikings'] = 2
>>> life.bigBang.stars.planets.mars.landers.vikings
2
>>> 'landers.vikings' in life.bigBang.stars.planets.mars
True
>>> life.get('bigBang.stars.planets.mars.landers.spirit', True)
True
>>> life.setdefault('bigBang.stars.planets.mars.landers.opportunity', True)
True
>>> 'landers.opportunity' in life.bigBang.stars.planets.mars
True
>>> life.bigBang.stars.planets.mars
{'landers': {'opportunity': True, 'vikings': 2}}
7
  • Thanks a lot, Mike. I added a get function which accepts dot notation (and a default value, as you noted) I think this new dotdictify class is going to make life a lot easier dealing with deeply nested dicts. Thanks so much.
    – Hal
    Sep 27, 2010 at 1:23
  • Do you need a get() function? What does it do that the existing get() doesn't? The get() you described in the question is equivalent to the get(key, None) that you get for free from dict. Sep 27, 2010 at 1:40
  • When I used the dotdictify class "as-is" w/ my Python 2.5 installation (Google App Engine SDK) the get function wasn't handling the dot notation requests for some reason. So I wrote a quick wrapper for the get() function to check for dot notation and, if so, pass to getattr (returning the default on exception), otherwise pass to dict.get(self, key, default)
    – Hal
    Sep 27, 2010 at 18:15
  • 1
    @Mike DeSimone I began using your dotdictify code and had a few problems when had I a dict with nested dicts that I dotdictified and I then tried to access the nested fields with dotted notation. The gist of the changes is here: gist.github.com/einnocent/8854896 If you are curious for a test case, please let me know.
    – einnocent
    Feb 6, 2014 at 23:51
  • Just made another change to my gist to accommodate when key is None -- yes, a bad edge case, but a real case nonetheless :)
    – einnocent
    Mar 4, 2014 at 19:42
6

The older answers have some pretty good tips in them, but they all require replacing standard Python data structures (dicts, etc.) with custom ones, and would not work with keys that are not valid attribute names.

These days we can do better, using a pure-Python, Python 2/3-compatible library, built for exactly this purpose, called glom. Using your example:

import glom

target = {}  # a plain dictionary we will deeply set on
glom.assign(target, 'a.b.c', {'d': 1, 'e': 2}, missing=dict)
# {'a': {'b': {'c': {'e': 2, 'd': 1}}}}

Notice the missing=dict, used to autocreate dictionaries. We can easily get the value back using glom's deep-get:

glom.glom(target, 'a.b.c.d')
# 1

There's a lot more you can do with glom, especially around deep getting and setting. I should know, since (full disclosure) I created it. That means if you find a gap, you should let me know!

4

To fellow googlers: we now have addict:

pip install addict

and

mapping.a.b.c.d.e = 2
mapping
{'a': {'b': {'c': {'d': {'e': 2}}}}}

I used it extensively.

To work with dotted paths, I found dotted:

obj = DottedDict({'hello': {'world': {'wide': 'web'}}})
obj['hello.world.wide'] == 'web'  # true
1
  • d = Dict({'a': {'ab': 'sa'}}) ; d['a.ab'] is giving empty dictionary with addict, am i doing something wrong ? Sep 9, 2020 at 7:44
2

I had used something similar in order to build somithing similar Trie for an application. I hope it helps.

class Trie:
    """
    A Trie is like a dictionary in that it maps keys to values.
    However, because of the way keys are stored, it allows
    look up based on the longest prefix that matches.

    """

    def __init__(self):
        # Every node consists of a list with two position.  In
        # the first one,there is the value while on the second
        # one a dictionary which leads to the rest of the nodes.
        self.root = [0, {}]


    def insert(self, key):
        """
        Add the given value for the given key.

        >>> a = Trie()
        >>> a.insert('kalo')
        >>> print(a)
        [0, {'k': [1, {'a': [1, {'l': [1, {'o': [1, {}]}]}]}]}]
        >>> a.insert('kalo')
        >>> print(a)
        [0, {'k': [2, {'a': [2, {'l': [2, {'o': [2, {}]}]}]}]}]
        >>> b = Trie()
        >>> b.insert('heh')
        >>> b.insert('ha')
        >>> print(b)
        [0, {'h': [2, {'a': [1, {}], 'e': [1, {'h': [1, {}]}]}]}]

        """

        # find the node to append the new value.
        curr_node = self.root
        for k in key:
            curr_node = curr_node[1].setdefault(k, [0, {}])
            curr_node[0] += 1


    def find(self, key):
        """
        Return the value for the given key or None if key not
        found.

        >>> a = Trie()
        >>> a.insert('ha')
        >>> a.insert('ha')
        >>> a.insert('he')
        >>> a.insert('ho')
        >>> print(a.find('h'))
        4
        >>> print(a.find('ha'))
        2
        >>> print(a.find('he'))
        1

        """

        curr_node = self.root
        for k in key:
            try:
                curr_node = curr_node[1][k]
            except KeyError:
                return 0
        return curr_node[0]

    def __str__(self):
        return str(self.root)

    def __getitem__(self, key):
        curr_node = self.root
        for k in key:
            try:
                curr_node = curr_node[1][k]
            except KeyError:
                yield None
        for k in curr_node[1]:
            yield k, curr_node[1][k][0]

if __name__ == '__main__':
    a = Trie()
    a.insert('kalo')
    a.insert('kala')
    a.insert('kal')
    a.insert('kata')
    print(a.find('kala'))
    for b in a['ka']:
        print(b)
    print(a)
0

Not a full-fledged solution, but a simple approach with no dependencies, and which doesn't require replacing/modifying the built-in dictionary type. Might fit the bill for some:

def get(nested_dict: dict, key: str):
    return reduce(lambda d, k: d[k], key.split('.'), nested_dict)

my_dict = {'a': {'b': {'c': 123}}}
get(my_dict, "a.b.c") # 123

The setter is not quite as nice, but works:

def set(nested_dict: dict, key: str, value):
    *keys, last_key = key.split('.')
    for k in keys:
        if k not in nested_dict:
            nested_dict[k] = dict()
        nested_dict = nested_dict[k]
    nested_dict[last_key] = value

set(my_dict, "very.very.many.levels", True)

A more full-fledged solution should probably check the keys accessed along the way. Probably other stuff I haven't though about at the moment.

0
pip install jmespath

Then

import jmespath

response = {"responseMetadata": {"HTTPStatusCode": 200 } }
response_code = jmespath.search('ResponseMetadata.HTTPStatusCode', response)

jmespath returns None if it can't find the key at the search path.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.