1

I am planning to use a large data set of hundreds of Gigbytes in Amazon S3/Athena and I have a question to experienced practitioners regarding the best practices for data storage (cost-wise and performance-wise).

My row data set contains about 40 columns with number-, date- and stringlike items. My queries are going to do some arithmetic operations and grouping quite frequently.

Is storing all data in form of strings a good idea? What would pros and cons to this approach? (introducing casting and converting the data types on the fly when querying)

or is it better to implement conversions straight away and store numbers in the numeric format, dates in timestmap formats etc.

1 Answer 1

1

I have two suggestions for you.

  1. I imagine you are already doing this, but use a columnar format such as Parquet to store your data. This will allow your queries to scan less data (assuming targeted queries) for your queries, which will make them more performant, and cost less, as Athena queries charge per TB of scanned data.

  2. In terms of column data types, I would use appropriate numeric types for numeric and date fields. When representing a number as a string, it consumes 1 or more bytes per digit (depending on encoding) in the number itself, which is not efficient when it comes to storing a numeric value. Take, for example, the number 203. As a string, it would require three bytes to encode it with UTF-8 (1 byte for '2', 1 byte for '0', and 1 byte for '3'). But the numeric value would fit into a single, unsigned byte.

You should see improvements in both cost and performance with both of these changes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.