I currently use this serialization library to save my data as Pair<double[], Double>
to a directory on my computer. The only data that is being serialized are numerical values in array form.
I've saved over 5 gigabytes of raw numerical data to that directory and I am expecting to continuously add more. I'm trying to future-proof the system to more efficiently store data (for use with very data hungry Neural Networks).
Is there a better way to store data rather than serializing Pair<double[], Double>
objects? Should I save everything to .txt or CSV files instead? I need to do everything I can do to minimize the memory footprint on my hard drive and reduce deserialization times. Any suggestions?