Questions tagged [hdf5]

The Hierarchical Data Format (HDF5) is a binary file format designed to store large amount of numerical data.

Filter by
Sorted by
Tagged with
0 votes
1 answer
19 views

What is the fastest way read slice from hdf5 dataset using h5py?

I have h5-file with dataset with dimensions 280000x62 and dtype float32. I need to read just 1 column, for example number 0. I do it with following code: def read_data_from_datasets(file_path, ...
Ilyas y.'s user avatar
  • 167
1 vote
1 answer
20 views

How to build h5py with MPI support (against Parallel HDF5) on Linux, 2023

I'm reproducing a deep-learning model on a linux server, which uses h5py to process data. However, following error occurs: Traceback (most recent call last): File "...", line ..., in ... ...
JamesNULLiu's user avatar
0 votes
1 answer
40 views

How to read an HDF5 compound datatype with unknown composition?

I have managed to read HDF5 data into rust using the hdf5 crate (shown below). But this method relies on knowing the exact structure of the dataset before hand. In the data I am trying to read the ...
TrippLamb's user avatar
  • 1,442
0 votes
1 answer
21 views

<HDF5 dataset "name": shape (), type "|f8"> in python

>>>fSM = h5py.File(filenames, 'r') >>>fSM.keys() <KeysViewHDF5 [Frequency', 'cycle']> >>>fSM['Frequency'] <HDF5 dataset "baseFrequency": shape (), type &...
lei su's user avatar
  • 1
0 votes
0 answers
15 views

How can I download hdf5 files from URLs in R?

I am trying to download a large number of HDF5 files to my computer from URLs provided by NASA's Earthdata Search. A download script (third tab) is provided by NASA, but since the rest of this project ...
cyanocitta34's user avatar
0 votes
1 answer
44 views

no hdf5 files with armadillo 12.6.2 on macos and cluster

I am trying to save an armadillo matrix (mat) to a hdf5 file. I am on a CentOS cluster and using anaconda without root privileges. Packages installed I have created an environment arma12 in which I ...
Keshav Das Agarwal's user avatar
0 votes
0 answers
53 views

How to compile fortran with HDF5 1.14.1 in Visual Studio 2019?

I am struggling to setup my fortran project properly. My project configuration is set to Release x64. When I put 'use hdf5' in my fortran module, I get: error #7002: Error in opening the compiled ...
hunch_bot's user avatar
0 votes
0 answers
32 views

h5py conflicts with swifter.apply

I am currently working with .h5 files. The file contains several tables, which I have to process (row filtering and other basic stuff). Then, as one of the steps, I have to compute an integral for ...
Stefano Cretti's user avatar
0 votes
0 answers
21 views

libhdf5.so.200 can't open shared object file: no such file or directory (repost)

I have installed pytorch and then nest-simulator in a conda virtual environment. When I import them, pytorch import fine but nest-simulator says ImportError: libhdf5.so.200: cannot open shared object ...
GT-R's user avatar
  • 27
0 votes
0 answers
29 views

How to create a 4d Matrix with Python and export in a SDF file?

I have written code in Python to create a 4d matrix (4 inputs, 1 output) that will be saved as an xlsx.file. Then this data is read in and saved to an SDF file. The 4d Matrix is a dummy matrix for ...
Alex's user avatar
  • 45
0 votes
1 answer
22 views

Linking in HDF5 to project using Matio fails

I am working on a QT 5 project and doing my development under MSYS2/Mingw64. Within that project is a sub-project using Matio, which is compiled into a static .a library. Matio, with the macros ...
Tyler Shellberg's user avatar
0 votes
3 answers
71 views

Python - HDF5 to numpy array - out of memory

I have a HDF5 file that contains 20'000'000 rows, each row has 8 float32 columns. The total raw memory size should be approx 640MB. I want to load this data in my Python app, however, during the load ...
Martin Perry's user avatar
  • 9,284
0 votes
1 answer
40 views

How to read hdf5 files with Python h5py package?

I'm new to coding and I want to know how to read a hdf5 file using Python. More specifically, I wish to access columns (by name) to extract information. I can read a h5 file using Pandas like this: pd....
ElenaM's user avatar
  • 1
0 votes
1 answer
19 views

How to check for available HDF5 options in CMake?

I'm using CMake to configure a project including HDF5 support. I can detect whether HDF5 is installed using find_package(HDF5 ...). According to the CMake manual entry, the find_package call will set ...
Dominic's user avatar
  • 452
0 votes
0 answers
66 views

'invalid option: --enable-cxx' after running 'brew install homebrew/science/hdf5 --enable-cxx'

I'm trying to run a cpp file using Rcpp. The error that keeps popping up is clang: error: unsupported option '-fopenmp'. After reading a bit, it seems to be the issue that I'm using Mac OS which does ...
Stan's user avatar
  • 1
0 votes
0 answers
20 views

How do I represent a multi-block cylindrical system in XDMF v2?

I can represent my HDF5 files using XDMF (specific type XMF used in CFD). But it is recognizing my R-Z coordinates and X-Y coordinates. It thinks I am working in 2D Cartesian X-Y. In other words, the ...
wander95's user avatar
  • 1,307
0 votes
1 answer
31 views

Write new dataset from 2d array in node-gdal-async

I am trying to create a new dataset from a 2d array in JavaScript. However, I can't seem to find my way around the docs to do this. To create a new dataset, I seem to have to use gdal.open in write ...
Kipnoedels's user avatar
  • 1,158
0 votes
2 answers
55 views

How do I iterate for all HDF5 files and save them as csv files

I am writing a python code that will loop over my SMAP HDF5 (10,000) files. I want to extract soil moisture rootzone and vegetation greenness. My code looks like this import os import tables import ...
Xei's user avatar
  • 11
6 votes
1 answer
946 views

PyTables install with python 3.11 fails on macOS M1

$ python -m pip install tables stops with Error: compiling Cython file Environment (I am within a virtual environment, created with pyenv. ) Only few packages installed atm Package Version ------...
theuema's user avatar
  • 107
0 votes
0 answers
16 views

VSCode Build Task Linker Issues with -lhdf5

I am trying to develop a build task in VSCode so that I can debug my code. When I compile the code using the following command: g++ -std=c++17 -o main main.cpp -I/usr/include/hdf5/serial -L/usr/lib/...
Ashamandarei's user avatar
0 votes
0 answers
57 views

Is not an HDF5 file error when iterating over multi-resolution HDF5 files

I want to run cooler balance (from cooler) on each resolution of each file with .mcool extension. for mcool_file in input/*.mcool; do while IFS= read -r id; do # id=./../input/A001C007.hg38.nodups....
melolili's user avatar
  • 1,289
0 votes
1 answer
143 views

installing HDF5 for C++

I'm trying to write a C++ program using HDF5, but I struggle with the build. It seems like cmake doesn't find the HDF5 library. Here is what I tried: main.cpp: #include <iostream> #include <...
marco's user avatar
  • 243
1 vote
1 answer
140 views

Convert multipage tiff file to hdf5

I would like to ask about conversion multipage tiff to h5 files. I could get 2-d data array from the image, but I have 21GB multipaged tiff container, so I need to convert the images to 3-d data shape....
Sean's user avatar
  • 11
1 vote
0 answers
47 views

use Python to Read/Write a large HDF5 file in remote server

I am currently maintaining a large data file (over 10G) that is updated daily on a Linux server. The data is in HDF5 format with groups and datasets. Now I have a local Windows client that needs to ...
agent1894's user avatar
0 votes
1 answer
24 views

Storing duplicate data in hdf5 files

I'm using H5py and numpy to manipulate HDF5 files. I want to expose two views of the same data, with different metadata. The approach I'm trying is to store a reference to the 1-dimenstional array ...
rtclay's user avatar
  • 93
0 votes
0 answers
17 views

HDF5: freeing memory chunks allocated by FAPL_CORE driver

I am creating in-memory HDF5 files (docs) like this: H5::FileAccPropList accPList=H5::FileAccPropList::DEFAULT; herr_t h5err=H5Pset_fapl_core(accPList.getId(),/* 4M chunks*/1<<22,/*backing_store*...
eudoxos's user avatar
  • 18.6k
1 vote
1 answer
77 views

How to combine entire content of several .h5 files into one large file using h5py?

I have several .h5 files (one file per day) each containing one dataset. I tried to combine them to one large .h5 file using h5py package with the following code: with h5py.File(combined_file_path, &...
vk21's user avatar
  • 11
0 votes
0 answers
27 views

|Solved| Damask3 insatll :Cannot open module file 'hdf5.mod' for reading at (1): No such file or directory

OK I solved the problem by using PETSC-3.14.5 The hdf5 environment always reports errors when I try to install damask3, but I have installed hdf5 in ubuntu-20.04, including download-hdf5-fortran-...
YX Liu's user avatar
  • 1
0 votes
0 answers
39 views

How manipulate transposed matrix on HDF5 Fortran

I am currently working on a code to solve a linear system of linear equations with Scalapack on Fortran. I am using HDF5 to pass the data, but I am having trouble because the answers do not match when ...
池田直哉's user avatar
3 votes
2 answers
142 views

Java - Read only parts of .mat file

I'm trying to read some portions of a .mat file, version v7.3, inside my Java application. In particular, I'd like to load only some parts of a variable stored inside the .mat file. Matlab supports ...
A M's user avatar
  • 831
0 votes
2 answers
88 views

HDF5 on Fortran what's inappropriate type

I try write HDF5 file on Fortran. I faced to error. I wrote example as follows. program ex11 ! 宣言 use, intrinsic :: iso_fortran_env use HDF5 implicit none intrinsic max, min ! ...
池田直哉's user avatar
-1 votes
1 answer
64 views

Storing data from device to main memory

I have a device vector that I continuously modify and then want to save in an HDF5 file. Because of the size of the device vector I cannot make multiple modifications and then save them to reduce the ...
Luluio's user avatar
  • 29
1 vote
2 answers
91 views

pandas.DataFrame.to_hdf() not saving attributes (metadata)

Is there a way to preserve the attributes of a pandas DataFrame when saving it to an hdf5 file? I would like to store metadata alongside a pandas DataFrame in an hdf5 file. Here is some simple code ...
quantumflash's user avatar
1 vote
1 answer
114 views

Open hdf5 dataframe in Julia

I saved a pandas dataframe with df.to_hdf("myfile.h5", key='my_df'). The file is composed of a bunch of numerical columns with 100 entries each. How can I open the same file in Julia as a ...
nop nop's user avatar
  • 93
1 vote
1 answer
24 views

How to resolve 'ValueError: invalid literal for int() with base 10: '5,3,2'' while reading hdf5 file using pandas?

I am trying to read a hdf5 file using Pandas, but getting this error: ValueError: invalid literal for int() with base 10: '5,3,2' I am only calling data = pd.read_hdf(path), so I have like no idea ...
Galedon's user avatar
  • 21
-1 votes
1 answer
57 views

Pandas dataframe - what causes this error?

My code: frame = pd.DataFrame({'a': np.random.randn(100)}) store = pd.HDFStore('mydata.h5') store['obj1'] = frame store['obj1_col'] = frame['a'] store.put('obj2',frame,foramt='table') store.select('...
Hatem Abdeen's user avatar
0 votes
0 answers
238 views

Mysterious "Segmentation fault" error when running a Fortran program

P.P.P.P.S Finally, I have found out the reason of the problem. Errors occurs on the variable vbuf, which is a Pointer type and its shape was defined previously. The shape of vbuf supposed to be (\...
Liren's user avatar
  • 19
0 votes
1 answer
53 views

Unable to create/save/load very large array on disk

For learning purposes I want to create ,save and then load(into tensorflow.keras) a very large int array of lengths in the order of 10^10. initially tried numpy but failed in the creation stage. ...
Vishnu Balaji's user avatar
0 votes
0 answers
23 views

Read a single column from a HDF5 dataframe in C

I have a HDF5 database and I need to load iteratively just one row per time. I'm having trouble understanding how to use the H5Sselect_hyperslab. There is a similar topic related to the h5py library ...
bartolus's user avatar
-3 votes
1 answer
90 views

Freeing memory allocated by hdf5 library

I'm running a c library that dynamically allocates memory and loads a string array from an h5 file. I get an error when freeing memory after use, I'm not sure if it's a windows problem or I'm not ...
piptin's user avatar
  • 365
0 votes
0 answers
44 views

Archive formats capable of appending data to a file

As in the title. I'm looking for an (uncompressed) archive format, which would allow for appending data to a file in the existing archive without unpacking a whole archive (or loading a whole file to ...
Kacper Rutkowski's user avatar
0 votes
0 answers
44 views

Error when saving an HDF5 file on a Julia server using JavaScript

I'm using JavaScript to select an HDF5 file on the client-side and send it to a Julia server. The Julia server is supposed to save the file and then read data from it. The read data should then be ...
lambcount's user avatar
1 vote
2 answers
102 views

How to choose whether to persist or discard changes in HDF5 file before closing

I'd like to manipulate a set of data in an hdf5 file and be able to decide, before closing the file, whether to discard every changes or not. From the doc of File drivers: HDF5 ships with a variety ...
Buzz's user avatar
  • 1,132
0 votes
0 answers
16 views

Slow write to hdf5 file

I have a problem of the following nature, there is an hdf5 pandas (PyTables). with open('vector.h5', 'wb') as f: f.close() vector = pd.HDFStore('vector.h5', mode='r+') with open('compare.h5', '...
Nikita's user avatar
  • 21
0 votes
0 answers
48 views

How to read partially corrupt hdf5 files with highfive in c++?

I'm using the header-only C++ HDF5 interface "highfive/H5Easy" to load my h5 files: auto file = H5Easy::File{path, H5Easy::File::ReadOnly}; auto p1= H5Easy::load<std::vector<float>&...
Michailina's user avatar
0 votes
1 answer
42 views

How to convert arary of byte of defaultdict to dictionary

I read from a h5 file with this code: f = h5py.File(fname, "r") data = f.get('table1')[...] the data is like this: array(b"defaultdict(<class 'list'>, {'B': [{'score': 4, 'skin': ...
ebrahim 's user avatar
2 votes
1 answer
80 views

Can you read HDF5 dataset directly into SharedMemory with Python?

I need to share a large dataset from an HDF5 file between multiple processes and, for a set of reasons, mmap is not an option. So I read it into a numpy array and then copy this array into shared ...
monday's user avatar
  • 319
0 votes
1 answer
76 views

Folders of images to hdf5 file

I am attempting to take folders of images and convert them to an hdf5 file type for use in a classification learning model. Each image in the folder should be paired with the name of the folder as its ...
TrinityCoding's user avatar
0 votes
0 answers
77 views

Unable to get data from hdf file in python

I have two hdf files, hdf4 and hdf5. In both cases, I am able to load the files in python using respective libraries (pyhdf and h5py). But inside the files, I can only read the data fields' name, size,...
Sky's user avatar
  • 9
1 vote
0 answers
162 views

How to handle large data with 2 HDF5 files, using h5py (read, write and calculations)

I'm developing a script where data needs to be analyzed, that is already available in an hdf5 file. This data can range from 10gb to a terabyte. I've created a script in Python, but the entire flow is ...
Makhaos's user avatar
  • 13

1
2 3 4 5
52