2

So I'm parsing data files into pandas dataframes and then saving the files to hdf5 using HDFStore. Code:

def convert_folder_hdf5(folder, save_loc=None):
    if save_loc is None:
        save_loc = folder

    filename = save_loc+'\\' + (folder.split('\\')[-1])+'.h5'
    store = pd.HDFStore(filename, format="table", complevel=9, complib='blosc')
    data = import_folder(folder)

    if data['voltage recording'] is not None:
        store['voltage_recording'] = data['voltage recording']
    if data['linescan'] is not None:
       store['linescan'] = data['linescan']

    store.root.attributes = data['file attributes']

    store.close()

However, I go to access store.root.attributes I'm given:

tables.exceptions.NoSuchNodeError: group ``/`` does not have a child named  ``attributes``

store.root returns (this particular folder has no linescan files):

/ (RootGroup) ''
  children := ['voltage_recording' (Group)]

Just working in the console I can manually do:

store.root.attributes = data['file attributes']

And then when I call store.root.attributes I get what I'm expecting to get.

However, if I then re-save the file, and re-open it what was set in store.root.attributes is gone (I get the same error as above).

1 Answer 1

1

You can't store directly in a node, but rather in the attributes section of the node. From here, you have to actually create a node and use the .attrs attribute to grab where you can actually store things.

Here's a way to do it directly:

In [16]: store.root._v_attrs.attributes = 'bar'

In [17]: store.root._v_attrs
Out[17]: 
/._v_attrs (AttributeSet), 5 attributes:
   [CLASS := 'GROUP',
    PYTABLES_FORMAT_VERSION := '2.1',
    TITLE := '',
    VERSION := '1.0',
    attributes := 'bar']

Pandas is still developing a more generic API to get/set attributes, feel free to comment: https://github.com/pydata/pandas/pull/7334

Further, specifying format='table' when opening the HDFStore is ignored. This ONLY is applicable on a to_hdf. You can't have a format for an entire store, just object-by-object.http://pandas.pydata.org/pandas-docs/stable/io.html#table-format

2
  • I'm a bit confused - did something change about how this works recently? I actually had a previous question about doing this, and your response then was similar to what I have in my code sample above. See: stackoverflow.com/questions/25232707/…
    – dan_g
    Sep 16, 2014 at 17:01
  • Not sure what I was thinking their. On a non-root node you can store (using .attrs). But this ONLY works with store.get_storer('df').attrs. Otherwise use the method above.
    – Jeff
    Sep 16, 2014 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.