My problem is that I have a very slow update query on a table with 14 million rows. I tried different things to tune my server which brought good performance but not for update queries.
I have two tables:
- T1 with 4 columns and 3 indexes on it (530 rows)
- T2 with 15 columns and 3 indexes on it (14 millions rows)
- I want to update the field vid (type integer) in T2 by the same value of vid in T1 by joining the two tables on a text field stxt.
Here is my query and its output:
explain analyse
update T2
set vid=T1.vid
from T1
where stxt2 ~ stxt1 and T2.vid = 0;
Update on T2 (cost=0.00..9037530.59 rows=2814247 width=131) (actual time=25141785.741..25141785.741 rows=0 loops=1) -> Nested Loop (cost=0.00..9037530.59 rows=2814247 width=131) (actual time=32.636..25035782.995 rows=679354 loops=1) Join Filter: ((T2.stxt2)::text ~ (T1.stxt1)::text) -> Seq Scan on T2 (cost=0.00..594772.96 rows=1061980 width=121) (actual time=0.067..5402.614 rows=1037809 loops=1) Filter: (vid= 1) -> Materialize (cost=0.00..17.95 rows=530 width=34) (actual time=0.000..0.069 rows=530 loops=1037809) -> Seq Scan on T1 (cost=0.00..15.30 rows=530 width=34) (actual time=0.019..0.397 rows=530 loops=1) Total runtime: 25141785.904 ms
As you can see the query took approximately 25141 seconds (~ 7 hours). f I understood well, the planner estimates the execution time to be 9037 seconds (~ 2.5 hours). Am I missing something here?
Here are information about my server config:
- CentOS 5.8, 20GB of RAM
- shared_buffers = 12GB
- work_mem = 64MB
- maintenance_work_mem = 64MB
- bgwriter_lru_maxpages = 500
- checkpoint_segments = 64
- checkpoint_completion_target = 0.9
- effective_cache_size = 10GB
I have run vacuum full and analyse several times on table T2 but this still does not improve much the situation.
PS: if I set full_page_writes to off, this improves considerably update queries, but I don't want to risk data loss. Do you please have any recommandations?
stxt2 like '%'||stxt1||'%'
. Both fields stxt are character varying. @radashk I tried this link but Postgres keeps telling meERROR: syntax error at or near "MERGE"
. How exactly should I try "MERGE"?