I have a large array of structs that I want to save to disk. The problem is that it takes a considerable amount of space. The following simple example reproduces the problem:
N = 1e5;
a = repmat(struct('x', zeros(100,1)), N, 1); % 90,400,064 Bytes in memory
save('testA.mat', 'a'); % 162 KB in disk
save('testB.mat', 'a', '-v7.3'); % 112 MB in disk
It seems compression efficiency is lost, probably due to the way how hdf5 works.
As my data is over 2 GB, I need to save the .mat in v7.3, but the saved file is too large (around 20x larger than expected).
Does anyone know a solution to save to .mat format that does not involve converting the array of structs into something else?
hdfview
to inspect the v7.3 file. My guess is it's storing each struct separately. As such, you can either change how you arrange the data (e.g. a struct of arrays, rather than an array of structs), or write the HDF file directly. I think there's no solution which you keep the data exactly as is, and create a v7.3 mat file with smaller size.