I have two Postgres tables: one normal table (20M rows), and one materialized view (2M rows). Both tables have an index on the "time" column. On both tables, I'm running a query which aggregates over a date range. When I run the query on the normal table, Postgres uses an Index Scan, which takes about 1 second. However, on the materialized view, Postgres uses a Bitmap Index Scan, which takes about 7 seconds. I'm not sure why it's not using a normal Index Scan here.
Both tables are roughly similar; the materialized view has 3 extra columns (2 floats and a boolean). The tables both have indices on their id
column and time
column; in the normal table, the id
column is a primary key. Both tables have been VACUUM ANALYZE
d.
The query is as follows:
SELECT count(*) as post_count,
sum(CASE WHEN in_reply_to_post_id IS NULL THEN 0 ELSE 1 END) as replies,
date_trunc('hour', time) as time_interval
FROM posts_enriched_materialized_2_weeks
WHERE time >= '2019-10-10' AND time < '2019-10-11'
GROUP BY time_interval;
The results of EXPLAIN ANALYZE
on the materialized view are as follows:
GroupAggregate (cost=264262.43..268387.60 rows=158191 width=32) (actual time=7794.743..7893.961 rows=24 loops=1)
Group Key: (date_trunc('hour'::text, "time"))
-> Sort (cost=264262.43..264691.99 rows=171822 width=46) (actual time=7790.080..7838.184 rows=175691 loops=1)
Sort Key: (date_trunc('hour'::text, "time"))
Sort Method: external merge Disk: 5464kB
-> Bitmap Heap Scan on posts_enriched_materialized_2_weeks (cost=4057.61..244033.53 rows=171822 width=46) (actual time=21.184..7672.166 rows=175691 loops=1)
Recheck Cond: (("time" >= '2019-10-10 00:00:00'::timestamp without time zone) AND ("time" < '2019-10-11 00:00:00'::timestamp without time zone))
Heap Blocks: exact=17420
-> Bitmap Index Scan on posts_enriched_materialized_2_weeks_time_index (cost=0.00..4014.65 rows=171822 width=0) (actual time=18.551..18.551 rows=175691 loops=1)
Index Cond: (("time" >= '2019-10-10 00:00:00'::timestamp without time zone) AND ("time" < '2019-10-11 00:00:00'::timestamp without time zone))
Planning time: 0.106 ms
Execution time: 7894.874 ms
EDIT: The results of EXPLAIN ANALYZE
on the non-MV table are as follows:
GroupAggregate (cost=193490.77..197635.89 rows=150641 width=32) (actual time=1168.018..1267.225 rows=24 loops=1)
Group Key: (date_trunc('hour'::text, "time"))
-> Sort (cost=193490.77..193943.19 rows=180969 width=46) (actual time=1163.293..1210.887 rows=175701 loops=1)
Sort Key: (date_trunc('hour'::text, "time"))
Sort Method: external merge Disk: 5472kB
-> Index Scan using posts_time_index on tweets (cost=0.44..172118.80 rows=180969 width=46) (actual time=0.900..1065.469 rows=175701 loops=1)
Index Cond: (("time" >= '2019-10-10 00:00:00'::timestamp without time zone) AND ("time" < '2019-10-11 00:00:00'::timestamp without time zone))
Planning time: 0.514 ms
Execution time: 1268.219 ms
Below is the correlation of the time
column for each table, requested in the comments. I don't exactly know what this value refers to, but it looks incredibly relevant:
posts 0.8844374
posts_enriched_materialized_2_weeks 0.09846322
Bitmap Index/Heap Scan
isn't what's slowing down your MV query. Rather, it's theexternal merge Disk: 5464kB
that takes most of the time. What's yourwork_mem
setting? Maybe you can increase it to at least 6MB?actual time
for the sort is 7838msec, and theBitmap Index Scan
only takes 18msecsum(CASE WHEN in_reply_to_post_id IS NULL THEN 0 ELSE 1 END)
with the equivalentcount(in_reply_to_post_id)
. Simpler and probably a bit faster.