1

I have a large array of structs that I want to save to disk. The problem is that it takes a considerable amount of space. The following simple example reproduces the problem:

N = 1e5;    
a = repmat(struct('x', zeros(100,1)), N, 1); % 90,400,064 Bytes in memory
save('testA.mat', 'a'); % 162 KB in disk
save('testB.mat', 'a', '-v7.3');  % 112 MB in disk

It seems compression efficiency is lost, probably due to the way how hdf5 works.

As my data is over 2 GB, I need to save the .mat in v7.3, but the saved file is too large (around 20x larger than expected).

Does anyone know a solution to save to .mat format that does not involve converting the array of structs into something else?

2
  • Try using hdfview to inspect the v7.3 file. My guess is it's storing each struct separately. As such, you can either change how you arrange the data (e.g. a struct of arrays, rather than an array of structs), or write the HDF file directly. I think there's no solution which you keep the data exactly as is, and create a v7.3 mat file with smaller size.
    – Justin
    Jun 9, 2020 at 21:30
  • The problem is that the array of structs is being used elsewhere, and changing the data format would affect other users and require the refactor of plenty of code. Seems the solution will be between implementing an ad-hoc save/load function; or change to a different data structure as you suggested.
    – serigado
    Jun 10, 2020 at 7:56

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.