Questions tagged [hdf5]
The Hierarchical Data Format (HDF5) is a binary file format designed to store large amount of numerical data.
2,598
questions
0
votes
1
answer
19
views
What is the fastest way read slice from hdf5 dataset using h5py?
I have h5-file with dataset with dimensions 280000x62 and dtype float32. I need to read just 1 column, for example number 0. I do it with following code:
def read_data_from_datasets(file_path, ...
1
vote
1
answer
20
views
How to build h5py with MPI support (against Parallel HDF5) on Linux, 2023
I'm reproducing a deep-learning model on a linux server, which uses h5py to process data.
However, following error occurs:
Traceback (most recent call last):
File "...", line ..., in ...
...
0
votes
1
answer
40
views
How to read an HDF5 compound datatype with unknown composition?
I have managed to read HDF5 data into rust using the hdf5 crate (shown below). But this method relies on knowing the exact structure of the dataset before hand.
In the data I am trying to read the ...
0
votes
1
answer
21
views
<HDF5 dataset "name": shape (), type "|f8"> in python
>>>fSM = h5py.File(filenames, 'r')
>>>fSM.keys()
<KeysViewHDF5 [Frequency', 'cycle']>
>>>fSM['Frequency']
<HDF5 dataset "baseFrequency": shape (), type &...
0
votes
0
answers
15
views
How can I download hdf5 files from URLs in R?
I am trying to download a large number of HDF5 files to my computer from URLs provided by NASA's Earthdata Search. A download script (third tab) is provided by NASA, but since the rest of this project ...
0
votes
1
answer
44
views
no hdf5 files with armadillo 12.6.2 on macos and cluster
I am trying to save an armadillo matrix (mat) to a hdf5 file. I am on a CentOS cluster and using anaconda without root privileges.
Packages installed
I have created an environment arma12 in which I ...
0
votes
0
answers
53
views
How to compile fortran with HDF5 1.14.1 in Visual Studio 2019?
I am struggling to setup my fortran project properly. My project configuration is set to Release x64. When I put 'use hdf5' in my fortran module, I get:
error #7002: Error in opening the compiled ...
0
votes
0
answers
32
views
h5py conflicts with swifter.apply
I am currently working with .h5 files. The file contains several tables, which I have to process (row filtering and other basic stuff). Then, as one of the steps, I have to compute an integral for ...
0
votes
0
answers
21
views
libhdf5.so.200 can't open shared object file: no such file or directory (repost)
I have installed pytorch and then nest-simulator in a conda virtual environment. When I import them, pytorch import fine but nest-simulator says ImportError: libhdf5.so.200: cannot open shared object ...
0
votes
0
answers
29
views
How to create a 4d Matrix with Python and export in a SDF file?
I have written code in Python to create a 4d matrix (4 inputs, 1 output) that will be saved as an xlsx.file.
Then this data is read in and saved to an SDF file. The 4d Matrix is a dummy matrix for ...
0
votes
1
answer
22
views
Linking in HDF5 to project using Matio fails
I am working on a QT 5 project and doing my development under MSYS2/Mingw64.
Within that project is a sub-project using Matio, which is compiled into a static .a library.
Matio, with the macros ...
0
votes
3
answers
71
views
Python - HDF5 to numpy array - out of memory
I have a HDF5 file that contains 20'000'000 rows, each row has 8 float32 columns. The total raw memory size should be approx 640MB.
I want to load this data in my Python app, however, during the load ...
0
votes
1
answer
40
views
How to read hdf5 files with Python h5py package?
I'm new to coding and I want to know how to read a hdf5 file using Python. More specifically, I wish to access columns (by name) to extract information.
I can read a h5 file using Pandas like this:
pd....
0
votes
1
answer
19
views
How to check for available HDF5 options in CMake?
I'm using CMake to configure a project including HDF5 support. I can detect whether HDF5 is installed using find_package(HDF5 ...). According to the CMake manual entry, the find_package call will set ...
0
votes
0
answers
66
views
'invalid option: --enable-cxx' after running 'brew install homebrew/science/hdf5 --enable-cxx'
I'm trying to run a cpp file using Rcpp. The error that keeps popping up is clang: error: unsupported option '-fopenmp'. After reading a bit, it seems to be the issue that I'm using Mac OS which does ...
0
votes
0
answers
20
views
How do I represent a multi-block cylindrical system in XDMF v2?
I can represent my HDF5 files using XDMF (specific type XMF used in CFD). But it is recognizing my R-Z coordinates and X-Y coordinates. It thinks I am working in 2D Cartesian X-Y. In other words, the ...
0
votes
1
answer
31
views
Write new dataset from 2d array in node-gdal-async
I am trying to create a new dataset from a 2d array in JavaScript. However, I can't seem to find my way around the docs to do this.
To create a new dataset, I seem to have to use gdal.open in write ...
0
votes
2
answers
55
views
How do I iterate for all HDF5 files and save them as csv files
I am writing a python code that will loop over my SMAP HDF5 (10,000) files. I want to extract soil moisture rootzone and vegetation greenness. My code looks like this
import os
import tables
import ...
6
votes
1
answer
946
views
PyTables install with python 3.11 fails on macOS M1
$ python -m pip install tables
stops with
Error: compiling Cython file
Environment
(I am within a virtual environment, created with pyenv. )
Only few packages installed atm
Package Version
------...
0
votes
0
answers
16
views
VSCode Build Task Linker Issues with -lhdf5
I am trying to develop a build task in VSCode so that I can debug my code. When I compile the code using the following command:
g++ -std=c++17 -o main main.cpp -I/usr/include/hdf5/serial -L/usr/lib/...
0
votes
0
answers
57
views
Is not an HDF5 file error when iterating over multi-resolution HDF5 files
I want to run cooler balance (from cooler) on each resolution of each file with .mcool extension.
for mcool_file in input/*.mcool; do
while IFS= read -r id; do # id=./../input/A001C007.hg38.nodups....
0
votes
1
answer
143
views
installing HDF5 for C++
I'm trying to write a C++ program using HDF5, but I struggle with the build. It seems like cmake doesn't find the HDF5 library. Here is what I tried:
main.cpp:
#include <iostream>
#include <...
1
vote
1
answer
140
views
Convert multipage tiff file to hdf5
I would like to ask about conversion multipage tiff to h5 files.
I could get 2-d data array from the image, but I have 21GB multipaged tiff container, so I need to convert the images to 3-d data shape....
1
vote
0
answers
47
views
use Python to Read/Write a large HDF5 file in remote server
I am currently maintaining a large data file (over 10G) that is updated daily on a Linux server. The data is in HDF5 format with groups and datasets.
Now I have a local Windows client that needs to ...
0
votes
1
answer
24
views
Storing duplicate data in hdf5 files
I'm using H5py and numpy to manipulate HDF5 files. I want to expose two views of the same data, with different metadata. The approach I'm trying is to store a reference to the 1-dimenstional array ...
0
votes
0
answers
17
views
HDF5: freeing memory chunks allocated by FAPL_CORE driver
I am creating in-memory HDF5 files (docs) like this:
H5::FileAccPropList accPList=H5::FileAccPropList::DEFAULT;
herr_t h5err=H5Pset_fapl_core(accPList.getId(),/* 4M chunks*/1<<22,/*backing_store*...
1
vote
1
answer
77
views
How to combine entire content of several .h5 files into one large file using h5py?
I have several .h5 files (one file per day) each containing one dataset. I tried to combine them to one large .h5 file using h5py package with the following code:
with h5py.File(combined_file_path, &...
0
votes
0
answers
27
views
|Solved| Damask3 insatll :Cannot open module file 'hdf5.mod' for reading at (1): No such file or directory
OK I solved the problem by using PETSC-3.14.5
The hdf5 environment always reports errors when I try to install damask3, but I have installed hdf5 in ubuntu-20.04, including download-hdf5-fortran-...
0
votes
0
answers
39
views
How manipulate transposed matrix on HDF5 Fortran
I am currently working on a code to solve a linear system of linear equations with Scalapack on Fortran. I am using HDF5 to pass the data, but I am having trouble because the answers do not match when ...
3
votes
2
answers
142
views
Java - Read only parts of .mat file
I'm trying to read some portions of a .mat file, version v7.3, inside my Java application.
In particular, I'd like to load only some parts of a variable stored inside the .mat file.
Matlab supports ...
0
votes
2
answers
88
views
HDF5 on Fortran what's inappropriate type
I try write HDF5 file on Fortran. I faced to error.
I wrote example as follows.
program ex11
! 宣言
use, intrinsic :: iso_fortran_env
use HDF5
implicit none
intrinsic max, min
!
...
-1
votes
1
answer
64
views
Storing data from device to main memory
I have a device vector that I continuously modify and then want to save in an HDF5 file. Because of the size of the device vector I cannot make multiple modifications and then save them to reduce the ...
1
vote
2
answers
91
views
pandas.DataFrame.to_hdf() not saving attributes (metadata)
Is there a way to preserve the attributes of a pandas DataFrame when saving it to an hdf5 file? I would like to store metadata alongside a pandas DataFrame in an hdf5 file.
Here is some simple code ...
1
vote
1
answer
114
views
Open hdf5 dataframe in Julia
I saved a pandas dataframe with df.to_hdf("myfile.h5", key='my_df').
The file is composed of a bunch of numerical columns with 100 entries each.
How can I open the same file in Julia as a ...
1
vote
1
answer
24
views
How to resolve 'ValueError: invalid literal for int() with base 10: '5,3,2'' while reading hdf5 file using pandas?
I am trying to read a hdf5 file using Pandas, but getting this error:
ValueError: invalid literal for int() with base 10: '5,3,2'
I am only calling data = pd.read_hdf(path), so I have like no idea ...
-1
votes
1
answer
57
views
Pandas dataframe - what causes this error?
My code:
frame = pd.DataFrame({'a': np.random.randn(100)})
store = pd.HDFStore('mydata.h5')
store['obj1'] = frame
store['obj1_col'] = frame['a']
store.put('obj2',frame,foramt='table')
store.select('...
0
votes
0
answers
238
views
Mysterious "Segmentation fault" error when running a Fortran program
P.P.P.P.S
Finally, I have found out the reason of the problem.
Errors occurs on the variable vbuf, which is a Pointer type and its shape was defined previously. The shape of vbuf supposed to be (\...
0
votes
1
answer
53
views
Unable to create/save/load very large array on disk
For learning purposes I want to create ,save and then load(into tensorflow.keras) a very large int array of lengths in the order of 10^10.
initially tried numpy but failed in the creation stage.
...
0
votes
0
answers
23
views
Read a single column from a HDF5 dataframe in C
I have a HDF5 database and I need to load iteratively just one row per time. I'm having trouble understanding how to use the H5Sselect_hyperslab.
There is a similar topic related to the h5py library ...
-3
votes
1
answer
90
views
Freeing memory allocated by hdf5 library
I'm running a c library that dynamically allocates memory and loads a string array from an h5 file.
I get an error when freeing memory after use, I'm not sure if it's a windows problem or I'm not ...
0
votes
0
answers
44
views
Archive formats capable of appending data to a file
As in the title.
I'm looking for an (uncompressed) archive format, which would allow for appending data to a file in the existing archive without unpacking a whole archive (or loading a whole file to ...
0
votes
0
answers
44
views
Error when saving an HDF5 file on a Julia server using JavaScript
I'm using JavaScript to select an HDF5 file on the client-side and send it to a Julia server. The Julia server is supposed to save the file and then read data from it. The read data should then be ...
1
vote
2
answers
102
views
How to choose whether to persist or discard changes in HDF5 file before closing
I'd like to manipulate a set of data in an hdf5 file and be able to decide, before closing the file, whether to discard every changes or not.
From the doc of File drivers:
HDF5 ships with a variety ...
0
votes
0
answers
16
views
Slow write to hdf5 file
I have a problem of the following nature, there is an hdf5 pandas (PyTables).
with open('vector.h5', 'wb') as f:
f.close()
vector = pd.HDFStore('vector.h5', mode='r+')
with open('compare.h5', '...
0
votes
0
answers
48
views
How to read partially corrupt hdf5 files with highfive in c++?
I'm using the header-only C++ HDF5 interface "highfive/H5Easy" to load my h5 files:
auto file = H5Easy::File{path, H5Easy::File::ReadOnly};
auto p1= H5Easy::load<std::vector<float>&...
0
votes
1
answer
42
views
How to convert arary of byte of defaultdict to dictionary
I read from a h5 file with this code:
f = h5py.File(fname, "r")
data = f.get('table1')[...]
the data is like this:
array(b"defaultdict(<class 'list'>, {'B': [{'score': 4, 'skin': ...
2
votes
1
answer
80
views
Can you read HDF5 dataset directly into SharedMemory with Python?
I need to share a large dataset from an HDF5 file between multiple processes and, for a set of reasons, mmap is not an option.
So I read it into a numpy array and then copy this array into shared ...
0
votes
1
answer
76
views
Folders of images to hdf5 file
I am attempting to take folders of images and convert them to an hdf5 file type for use in a classification learning model. Each image in the folder should be paired with the name of the folder as its ...
0
votes
0
answers
77
views
Unable to get data from hdf file in python
I have two hdf files, hdf4 and hdf5. In both cases, I am able to load the files in python using respective libraries (pyhdf and h5py). But inside the files, I can only read the data fields' name, size,...
1
vote
0
answers
162
views
How to handle large data with 2 HDF5 files, using h5py (read, write and calculations)
I'm developing a script where data needs to be analyzed, that is already available in an hdf5 file. This data can range from 10gb to a terabyte.
I've created a script in Python, but the entire flow is ...