11

I've been working on a project that is generating on the order of 10 - 100 million outputs from a simulation that I would like to store for future analyses. There are several nature levels of organization to the data e.g. Classrooms hold Students who take Tests which have a handful of different performance metrics.

It seems like my data is border line in terms of being able to fit in memory all at once (given the calculation of the simulations requires a fair amount of data in memory to do the calculation), but I don't have any immediate need for all of the data to be available to my program at once.

I am considering whether it would be better to be outputting the calculated values to a SQL database or a flat text file. I am looking for advice about which approach might be faster/easier to maintain (or if you have an alternate suggestion for storing the data I am open to that).

I don't need to be able to share the data with anyone else or worry about accessing the data years down the line. I just need a convenient way to avoid regenerating the simulations everytime I want to carry out a tweak to the analysis of the values.

4 Answers 4

4

I'd consider using a database - 100 million files is too many for a file system without some kind of classification scheme, while a database can easily handle this many rows. You could just serialize the output into a BLOB column so you don't have to map it. Also, consider that SQL Server has file stream access so this could be essentially a hybrid approach where SQL manages the files for you.

3

Offhand, it sounds like you would be better off saving the results of each simulation run into a flat file. It need not be a text file - it could be binary.

After one or more simulation runs, the files could be read and placed into a data warehouse for later analysis.

1
  • 2
    If the data are in .net objects, Binary Serialization might work magically.
    – SWeko
    Dec 21, 2012 at 1:35
2

The back-of-the-envelope rate for loading the data from an RDBMS server into memory is roughly 10K records per second. If you have 100M records, and if you must use all data at some point, you are looking at roughly three hours to load the data. That is before you do any calculations!

Plain files can be orders of magnitude faster. You can get pretty fast with a text-based file; going binary would improve your speed some more at the expense of readability of your data file.

2

Take a look at MongoDB, which is around 30x-50x faster in performance than SQL Server 2008 apparently.

http://blog.michaelckennedy.net/2010/04/29/mongodb-vs-sql-server-2008-performance-showdown/

1
  • MongoDB also supports files if you wish to use files.
    – PmanAce
    Dec 21, 2012 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.