3

I have a data structure which is essentially a lookup for some calculations that take a really long time (100ms) to calculate and need to be used over and over. I have roughly 6,000,000 of these calculations and want to load them into memory when my application starts (I will precalculate them all).

The question is can I store this as a memory mapped file (dictionary of something) or should I store it in a db and then load it into ram on program start up? How fast would binary serialization be?

What are my options?

2
  • Please note that you don't need "C# .NET" in your title, as that's what tags are for. Aug 9, 2011 at 0:05
  • There's nothing that suggests in your question that simply storing them in an array or List<> won't be the best solution. Six million tends to fit, everything else you're contemplating will just make it slower. Aug 9, 2011 at 0:25

2 Answers 2

3

Binary serialization is fast, specially if you only have to load it once. Load speed from a database really depends on how the data is structured. The advantage to using a database is easy of management. If you want to easily manage, change, track changes, or use these values with multiple clients, then a DB would be the way to go. If they are never going to change, a file would suffice.

2

You have to try different approaches and measure for yourself. There is no other road to solve performance problems. Note that you need to have some concreate goal in mind (like 1 second for load/1ms for lookup).

Options:

  • calculate all values on startup and sotre in some lookup (preallocated array/dictionary)
  • calculate on demand and sotre in some lookup (preallocated array/dictionary)
  • calculate in advance and load uncompressed (note that amount of data you have is large and will take noticable 1-3 seconds time to load)
  • calculate in advance and load on demand
  • calculate in advance and load compressed data

I'd recommned trying to compute all values on load and see if it works fast enough - most likely easiest way to go.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.