I'm building an Amazon Redshift data warehouse, and experiencing unexpected performance impacts based on the defined size of the VARCHAR column. Details are as follows. Three of my columns are shown from pg_table_def:
schemaname | tablename | column | type | encoding | distkey | sortkey | notnull
------------+-----------+-----------------+-----------------------------+-----------+---------+---------+---------
public | logs | log_timestamp | timestamp without time zone | delta32k | f | 1 | t
public | logs | event | character varying(256) | lzo | f | 0 | f
public | logs | message | character varying(65535) | lzo | f | 0 | f
I've recently run Vacuum and Analyze, I have about 100 million rows in the database, and I'm seeing very different performance depending on which columns I include.
Query 1: For instance, the following query takes about 3 seconds:
select log_timestamp from logs order by log_timestamp desc limit 5;
Query 2: A similar query asking for more data runs in 8 seconds:
select log_timestamp, event from logs order by log_timestamp desc limit 5;
Query 3: However, this query, very similar to the previous, takes 8 minutes to run!
select log_timestamp, message from logs order by log_timestamp desc limit 5;
Query 4: Finally, this query, identical to the slow one but with explicit range limits, is very fast (~3s):
select log_timestamp, message from logs where log_timestamp > '2014-06-18' order by log_timestamp desc limit 5;
The message
column is defined to be able to hold larger messages, but in practice it doesn't hold much data: the average length of the message field is 16 charachters (std_dev 10). The average length of the event field is 5 charachters (std_dev 2). The only distinction I can really see is the max length of the VARCHAR field, but I wouldn't think that should have an order of magnitude affect on the time a simple query takes to return!
Any insight would be appreciated. While this isn't the typical use case for this tool (we'll be aggregating far more than we'll be inspecting individual logs), I'd like to understand any subtle or not-so-subtle affects of my table design.
Thanks!
Dave