3

I have a data structure that consists of a three-level nested dict that keeps counts of occurrences of a three part object. I'd like to build a DataFrame out of it with a specific shape, but I can't figure out a way to do it that doesn't involve consuming a lot of working memory---because the table is quite large (several GBs at full extent).

The basic functionality looks like this:

class SparseCubeTable:
    def __init__(self):
        self.table = {}
        self.dim1 = []
        self.dim2 = []
        self.dim3 = []

    def increment(self, dim1, dim2, dim3):
        if dim1 in self.table:
            if dim2 in self.table[dim1]:
                if dim3 in self.table[dim1][dim2]:
                    self.table[dim1][dim2][dim3] += 1
                else:
                    self.dim3.append(dim3)
                    self.table[dim1][dim2][dim3] = 1
            else:
                self.dim2.append(dim2)
                self.dim3.append(dim3)
                self.table[dim1][dim2] = {dim3:1}
        else:
            self.dim1.append(dim1)
            self.dim2.append(dim2)
            self.dim3.append(dim3)
            self.table[dim1] = {dim2:{dim3:1}}

This was constructed to make summing over keys easier, among other things. A SparseCubeTable is used like this:

In [23]: example = SparseCubeTable()

In [24]: example.increment("thing1", "thing2", "thing3")

In [25]: example.increment("thing1", "thing2", "thing3")

In [26]: example.increment("thing4", "thing5", "thing6")

In [27]: example.increment("thing1", "thing3", "thing5")

And you can get the data like this:

In [29]: example.table['thing1']['thing2']['thing3']
Out[29]: 2

The sort of DataFrame I want looks like this:

1 2 3 4
thing1 thing2 thing3 2
thing1 thing3 thing5 1
thing4 thing5 thing6 1

The DataFrame is going to be saved as an HDF5 db with columns 1-3 indexed and statistical transformations on column 4 (that require the whole table be temporarily in memory).

The problem is that the pandas.DataFrame.from_dict function builds a whole other sort of structure with the keys used as row labels, as far as I understand it. However, trying to use from_records forces me to copy out the whole data structure into a list, meaning that I now have double the memory size to worry about.

I tried implementing the solution in:

Create a pandas DataFrame from generator?

but in 0.12.0 what it ends up doing is first building a giant list of strings which is even worse. I assume writing out the structure to a csv and reading it back in is also going to be terrible on memory.

Is there a better way of doing this? Or should I just try to squeeze memory even further in the SparseCubeTable somehow? It seems so wasteful to have to build an intermediate list data structure to use from_records.

8
  • I only took time to skim, but I think the orient keyword on from_dict might make it work for you.
    – Dan Allan
    Commented Dec 10, 2013 at 15:23
  • you might be able to build a Panel directly, Panel(example.table), which might work for you
    – Jeff
    Commented Dec 10, 2013 at 15:30
  • @DanAllan Tried that. It just takes one level of keys and then makes them column or row indices. I have this tripartite structure instead. Commented Dec 10, 2013 at 15:33
  • @Jeff I am fiddling around with your Panel solution, but I need to do group-bys along all three columns (of my DataFrame version), and Panel.group_by doesn't exist and Panel.to_frame() creates an empty DataFrame. Commented Dec 10, 2013 at 15:42
  • I was going to suggest, you don't go this route at all. Instead build a HDF table first directly. Read it in chunks, and do value_counts, write to a new table and repeat, see this solution: stackoverflow.com/questions/15798209/….
    – Jeff
    Commented Dec 10, 2013 at 15:52

1 Answer 1

5

Here is a code for an efficient solution.

Create some data looking like yours. This is a list of 1000 3-tuples

In [1]: import random

In [2]: tags = [ 'thing{0}'.format(i) for i in xrange(100) ]

In [3]: data = [ (random.choice(tags),random.choice(tags),random.choice(tags)) for i in range(1000) ]

Our writing function, makes sure that when we write the index is globally unique (its not actually necessary, but since the index is actually written its 'nicer')

In [4]: def write(store,c):
   ...:     df = DataFrame(c,columns=['dim1','dim2','dim3'])
   ...:     try:
   ...:         nrows = store.get_storer('df').nrows
   ...:     except:
   ...:         nrows = 0
   ...:     df.index += nrows
   ...:     store.append('df',df,data_columns=True)
   ...:     return []
   ...: 

In [5]: collector = []

In [6]: store = pd.HDFStore('data.h5',mode='w')

Iterate thru your data (or from a stream or whatever), and write it.

In [7]: for i, d in enumerate(data):
   ...:     collector.append(d)
   ...:     if i % 100 == 0 and i:
   ...:         collector = write(store,collector)
   ...:         

In [8]: write(store,collector)
Out[8]: []

The store

In [9]: store
Out[9]: 
<class 'pandas.io.pytables.HDFStore'>
File path: data.h5
/df            frame_table  (typ->appendable,nrows->1000,ncols->3,indexers->[index],dc->[dim1,dim2,dim3])
In [9]: store
Out[9]: 
<class 'pandas.io.pytables.HDFStore'>
File path: data.h5
/df            frame_table  (typ->appendable,nrows->1000,ncols->3,indexers->[index],dc->[dim1,dim2,dim3])

In [10]: store.select('df')
Out[10]: 
       dim1     dim2     dim3
0   thing28  thing87  thing29
1   thing62  thing70  thing50
2   thing64  thing12  thing98
3   thing33  thing98  thing46
4   thing46   thing5  thing76
5    thing2   thing9  thing21
6    thing1  thing63  thing68
7   thing42  thing30  thing45
8   thing56  thing71  thing77
9   thing99  thing10  thing91
10  thing40   thing9  thing10
11  thing70  thing54  thing59
12  thing94  thing65   thing3
13  thing93  thing24  thing25
14  thing95  thing94  thing86
15  thing41  thing55   thing3
16  thing88  thing10  thing47
17  thing89  thing58  thing33
18  thing16  thing66  thing55
19  thing68  thing20  thing99
20  thing34  thing71  thing28
21  thing67  thing87  thing97
22  thing77  thing74   thing6
23  thing63  thing41  thing30
24  thing14  thing62  thing66
25  thing20  thing36  thing67
26  thing33  thing19  thing58
27   thing0  thing71  thing24
28   thing1  thing48  thing42
29  thing18  thing12   thing4
30  thing85  thing97  thing20
31  thing73  thing71  thing70
32  thing91  thing43  thing48
33  thing45   thing6  thing87
34   thing0  thing28   thing8
35  thing56  thing38  thing61
36  thing39  thing92  thing35
37  thing69  thing26  thing22
38  thing16  thing16  thing79
39   thing4  thing16  thing12
40  thing81  thing79   thing1
41  thing77  thing90  thing83
42  thing53  thing17  thing89
43  thing53  thing15  thing37
44  thing25   thing7  thing20
45  thing44  thing14  thing25
46  thing62  thing84  thing23
47  thing83  thing50  thing60
48  thing68  thing64  thing24
49  thing73  thing53  thing43
50  thing86  thing67  thing31
51  thing75  thing63  thing82
52   thing8  thing10  thing90
53  thing34  thing23  thing12
54  thing66  thing97  thing26
55  thing66  thing53  thing27
56  thing79  thing22  thing37
57  thing43  thing82  thing66
58  thing87  thing53  thing92
59  thing33  thing71  thing97
        ...      ...      ...

[1000 rows x 3 columns]

In [11]: store.close()

Then you can do interesting things. If you are not reading the entire set in you may want to chunk this (which is a bit more involved if you are counting things).

In [56]: pd.read_hdf('data.h5','df').apply(lambda x: x.value_counts())
Out[56]: 
         dim1  dim2  dim3
thing0     12     6     8
thing1     14     7     8
thing10    10    10     7
thing11     8    10    14
thing12    11    14    11
thing13    11    12     7
thing14     8    14     3
thing15    12    11    11
thing16     7    10    11
thing17    16     9    13
thing18    13     8    10
thing19    11     7     8
thing2      9     5    17
thing20     6     7    11
thing21     7     8     8
thing22     4    17    14
thing23    14    11     7
thing24    10     5    14
thing25    11    11    12
thing26    13    10    15
thing27    12    15    16
thing28    11    10     8
thing29     7     7     8
thing3     11    14    14
thing30    11    16     8
thing31     7     6    12
thing32     8    12     9
thing33    13    12    12
thing34    12     8     5
thing35     6    10     8
thing36     6     9    13
thing37     8    10    12
thing38     7    10     4
thing39    14    11     7
thing4      9     7    10
thing40    12     8     9
thing41     8    16    11
thing42     9    11    13
thing43     8     6    13
thing44     9    13    11
thing45     7    13     7
thing46    12     8    13
thing47     9    10     9
thing48     8     9     9
thing49     4     8     7
thing5     13     7     7
thing50    14    12     9
thing51     5     7    11
thing52     9    11    12
thing53     9    15    15
thing54     7     9    13
thing55     6    10    10
thing56    12    11    11
thing57    12     9    11
thing58    12    12    10
thing59     6    13    10
thing6      8     5     7
thing60    12     9     6
thing61     5     9     9
thing62     8    10     8
          ...   ...   ...

[100 rows x 3 columns]

You can then do a 'groupby' like this:

In [69]: store = pd.HDFStore('data.h5')

In [61]: dim1 = Index(store.select_column('df','dim1').unique())
In [66]: store.close()

In [67]: groups = dim1[0:10]

In [68]: groups
Out[68]: Index([u'thing28', u'thing62', u'thing64', u'thing33', u'thing46', u'thing2', u'thing1', u'thing42', u'thing56', u'thing99'], dtype='object')

In [70]: pd.read_hdf('data.h5','df',where='dim1=groups').apply(lambda x: x.value_counts())
Out[70]: 
         dim1  dim2  dim3
thing1     14     2     1
thing10   NaN     1     1
thing11   NaN     1     2
thing12   NaN     5   NaN
thing13   NaN     1   NaN
thing14   NaN     1     1
thing15   NaN     1     1
thing16   NaN     1     3
thing17   NaN   NaN     2
thing18   NaN     1     1
thing19   NaN     1     2
thing2      9     1     1
thing20   NaN     2   NaN
thing21   NaN   NaN     1
thing22   NaN     2     2
thing23   NaN     2     3
thing24   NaN     2     1
thing25   NaN     3     2
thing26   NaN     2     2
thing27   NaN     3     1
thing28    11   NaN   NaN
thing29   NaN     1     2
thing30   NaN     2   NaN
thing31   NaN     1     1
thing32   NaN     1     1
thing33    13     1     2
thing34   NaN     1   NaN
thing35   NaN     1   NaN
thing36   NaN     1     1
thing37   NaN     1     2
thing38   NaN     3   NaN
thing39   NaN     3     1
thing4    NaN     2   NaN
thing41   NaN   NaN     1
thing42     9     1     1
thing43   NaN   NaN     1
thing44   NaN     1     2
thing45   NaN   NaN     2
thing46    12   NaN     1
thing47   NaN     1     1
thing48   NaN     1   NaN
thing49   NaN     1   NaN
thing5    NaN     2     2
thing50   NaN   NaN     3
thing51   NaN     2     2
thing52   NaN     1     3
thing53   NaN     2     4
thing55   NaN   NaN     2
thing56    12     1     1
thing57   NaN   NaN     3
thing58   NaN     1     2
thing6    NaN   NaN     1
thing60   NaN     1     1
thing61   NaN     1     4
thing62     8     2     1
thing63   NaN     1     1
thing64    15   NaN     1
thing66   NaN     1     2
thing67   NaN     2   NaN
thing68   NaN     1     1
          ...   ...   ...

[90 rows x 3 columns]
3
  • Ah, this is really nice. I'm not sure I quite understand the last bit: "If you are not reading the entire set in you may want to chunk this." I do ultimately need the count of "thing2 thing9 thing21" together since the whole point of the exercise is to compute certain kinds of statistics on the triples, although I certainly need the other value_counts too. Commented Dec 10, 2013 at 17:10
  • what I mean is this. If you can read the store HDF5 table all in memory, it becomes easy to do things (e.g. like the value counts above, which btw I am not sure is exactly what you want, but a way to start). If you cannot read the HDF5 table in memory, you may want to work on it in chunks, e.g. essentially operate on parts that you select, say via a subset of groups. (which you select like above).
    – Jeff
    Commented Dec 10, 2013 at 17:13
  • Ah, I see what you mean. I suspect the table is too large to hold in memory. Building it in chunks means that I have to build something of size similar to the SparseCubeTable in my original example, so that I can get not just the individual per-token value_counts, but the actual counts of the number of times that "thing1 thing2 thing3" happens together in the original XML files, proportionate to all the other "thingA thingB thingC" in the data. In which case I would be back to my original problem. BTW, not everything can happen in each dimension, which is why this is so complicated. Commented Dec 11, 2013 at 1:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.