6

I have never worked with HDF5 files before, and to get started I received some example files. I've been checking out all the basics with h5py, looking at the different groups in these files, their names, keys, values and so on. Everything works fine, until I want to look at the datasets that are saved in the groups. I get their .shape and .dtype, but when I try accessing a random value by indexing (e.g. grp["dset"][0]), I get the following error:

IOError                                   Traceback (most recent call last)
<ipython-input-45-509cebb66565> in <module>()
      1 print geno["matrix"].shape
      2 print geno["matrix"].dtype
----> 3 geno["matrix"][0]

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/dataset.pyc in __getitem__(self, args)
    443         mspace = h5s.create_simple(mshape)
    444         fspace = selection._id
--> 445         self.id.read(mspace, fspace, arr, mtype)
    446
    447         # Patch up the output for NumPy

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/h5d.so in h5py.h5d.DatasetID.read (h5py/h5d.c:2782)()

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_proxy.so in h5py._proxy.dset_rw (h5py/_proxy.c:1709)()

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_proxy.so in h5py._proxy.H5PY_H5Dread (h5py/_proxy.c:1379)()

IOError: Can't read data (Can't open directory)

I've posted this problem in the h5py Google group, where it was suggested that there might be a filter on the dataset I don't have installed. But the HDF5 file was created using only gzip compression, which should be a portable standard, as far as I understood.
Does someone know what I might be missing here? I can't even find a description of this error or similar problems anywhere, and the file, including the problematic dataset, can be easily opened with the HDFView software.

Edit
Apparently, this error occurs because, for some reason, the gzip compression filter is not available on my system. If I try to create an example file with gzip compression, this happens:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-33-dd7b9e3b6314> in <module>()
      1 grp = f.create_group("subgroup")
----> 2 grp_dset = grp.create_dataset("dataset", (50,), dtype="uint8", chunks=True, compression="gzip")

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/group.pyc in create_dataset(self, name, shape, dtype, data, **kwds)
     92         """
     93 
---> 94         dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
     95         dset = dataset.Dataset(dsid)
     96         if name is not None:

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/dataset.pyc in make_new_dset(parent, shape, dtype, data, chunks, compression, shuffle, fletcher32, maxshape, compression_opts, fillvalue, scaleoffset, track_times)
     97 
     98     dcpl = filters.generate_dcpl(shape, dtype, chunks, compression, compression_opts,
---> 99                   shuffle, fletcher32, maxshape, scaleoffset)
    100 
    101     if fillvalue is not None:

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/filters.pyc in generate_dcpl(shape, dtype, chunks, compression, compression_opts, shuffle, fletcher32, maxshape, scaleoffset)
    101 
    102         if compression not in encode:
--> 103             raise ValueError('Compression filter "%s" is unavailable' % compression)
    104 
    105         if compression == 'gzip':

ValueError: Compression filter "gzip" is unavailable

Does anyone have experience with that? The installation of the HDF5 library as well as the h5py package didn't seem to go wrong...

8
  • That looks like an incorrect path/permissions problem to me.
    – aruisdante
    Oct 10, 2014 at 14:08
  • 1
    Is dset among the dataset names listed by grp.keys()? Do you have a sample file you can post somewhere for us to see? Also consider doing h5ls thefile on the command line and see what that shows. Oct 10, 2014 at 14:11
  • Please could you post the output of print geno["matrix"]?
    – Yossarian
    Oct 11, 2014 at 18:20
  • @JohnZwinck Yes, the dataset appears in the groups keys. The h5ls command lists the two groups that are in the file. @Yossarian The output of just print geno["matrix"] is <HDF5 dataset "matrix": shape (1008, 11623), type "|u1">. Oct 13, 2014 at 8:21
  • 1
    I tried creating an example file, but couldn't (see my edited question). Apparently the whole problem is that I'm missing the gzip compression filter, but I don't understand how this could have happened. Oct 13, 2014 at 12:44

5 Answers 5

9

This error happens when h5py can't find the plugin it needs to open the file. For many of the common plugins, this can be solved by adding:

import hdf5plugin

before you use the h5py library. You don't have to use the hdf5plugin library directly, you just import it. Different imports may work for you depending on which plugin the file is using - it would be helpful if this error message were a bit more descriptive.

2
  • 1
    had the same issue. This suggestion fixed it
    – Kasrel
    Oct 1, 2021 at 12:27
  • this is crazy that you have to do this!!!
    – rump roast
    Oct 6, 2022 at 20:07
3

I had the same problem. I solved it with

import tables

and it works fine now

1

Can't just comment - reputation too low.

I had the same issue, simply ran "conda update anaconda" and the problem is gone.

1
  • 1
    That makes sense. Over at the h5py Google group, we figured that there was something wrong with the Anaconda h5py version, and they fixed it there. Thanks for reminding me to close the issue here as well! Dec 12, 2014 at 12:51
1

I had a similar issue,

$ python3 -c 'import h5py; f=h5py.File("file.h5"); d=f["FVC"][:,:]'                                                               

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)
  File "/home/pinaultf/system/anaconda2/envs/deveg-dev/lib/python3.5/site-packages/h5py/_hl/dataset.py", line 482, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read (/home/ilan/minonda/conda-bld/work/h5py/h5d.c:3240)
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw (/home/ilan/minonda/conda-bld/work/h5py/_proxy.c:1869)
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread (/home/ilan/minonda/conda-bld/work/h5py/_proxy.c:1517)
OSError: Can't read data (Can't open directory)

I had this problem in one virtual environment and not in another, even if apparently the h5py version was the same (2.6.0).

I solved this issue with :

$ pip uninstall h5py
$ pip install h5py
-1

I switched python 3.6 to python 3.8 and 'import tables', solved.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.