0

I receive more than 1 million rows per hour, I need a way to save them for later processing.

Write performance is the most important factor here but I don't know which is the best way to handle this much data.

Should I save to XML, or will a SQL Server DB perform that fast?

Should I split the XML over multiple files so that I can process them in batches or should I have just one very large file?

I would ideally like to use a SQL Server DB because it will make the processing easier, however I do not know if it will be able to handle the amount of data I will be dealing with.

The data I am trying to save is:

// TODO: Data Structure here
1
  • How much data are you dealing with per row? if you don't know this, please post your SQL table schema that you would use if you were using SQL Server. Would the SQL DB be stored on the same server as the processing? or would the SQL Server be a separate server, where as the XML file would be direct to a local disk.
    – Seph
    Jan 30, 2012 at 5:21

1 Answer 1

0

I usually like databases over writing files, espcially if you are getting that many rows in an hour. I assume it would be 1 million separate files an hour?

Databases are pretty flexible and are a lot easier to get data from vs a bajillion xml files.

If the database ever gets too big, where you are afraid of performance, you can archive some of the older data and remove it from the main database.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.